Oct 30 16:46:19.957653 ip-10-0-135-153 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:46:20.322742 ip-10-0-135-153 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:46:20.322742 ip-10-0-135-153 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:46:20.322742 ip-10-0-135-153 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:46:20.322742 ip-10-0-135-153 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:46:20.327448 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327339 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:20.327448 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327444 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:20.327448 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327449 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:20.327448 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327453 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:20.327448 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327457 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327461 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327465 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327470 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327474 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327478 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327482 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327485 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327489 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327493 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327497 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327501 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327505 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327509 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327518 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327523 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327527 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327532 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327536 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327540 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:20.327741 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327544 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327547 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327551 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327555 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327558 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327563 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327567 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327570 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327574 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327579 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327583 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327587 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327591 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327596 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327603 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327616 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327621 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327625 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327630 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327634 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:20.328568 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327639 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327643 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327647 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327651 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327656 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327660 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327664 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327669 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327673 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327677 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327681 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327685 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327690 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327694 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327698 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327702 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327706 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327711 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327715 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:20.329456 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327719 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327723 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327727 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327732 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327736 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327740 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327744 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327752 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327760 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327773 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327778 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327783 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327787 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327791 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327796 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327800 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327804 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327808 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327812 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:20.330223 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327817 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327821 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327826 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327830 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327834 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327838 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327842 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327846 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327851 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327855 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327859 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327863 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327868 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327872 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327876 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327880 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327885 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327889 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327898 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:20.331051 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327905 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327910 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327914 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327918 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.327924 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329233 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329243 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329247 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329252 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329256 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329261 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329265 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329270 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329275 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329279 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329283 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329287 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329291 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329295 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:20.331612 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329300 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329304 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329308 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329312 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329317 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329321 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329326 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329330 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329334 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329338 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329342 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329346 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329350 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329371 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329376 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329381 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329386 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329390 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329396 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329400 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:20.332148 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329404 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329409 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329413 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329417 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329421 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329426 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329430 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329435 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329439 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329444 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329448 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329452 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329457 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329461 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329465 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329469 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329473 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329478 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329483 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:20.333050 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329487 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329491 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329496 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329500 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329504 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329509 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329513 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329517 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329522 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329526 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329532 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329536 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329542 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329546 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329550 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329554 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329558 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329562 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329567 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329571 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:20.333884 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329575 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329579 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329583 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329587 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329592 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329596 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329600 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329604 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329611 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329617 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329625 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329630 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329634 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329638 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329642 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329647 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329652 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329656 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329660 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:20.334585 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329665 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329669 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329673 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329678 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329682 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329686 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329691 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329696 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329702 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329707 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329712 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329716 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329720 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.329725 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330574 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330583 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330588 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330593 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330598 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:20.335184 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330603 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330607 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330611 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330616 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330621 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330625 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330630 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330634 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330638 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330643 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330647 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330651 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330656 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330660 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330665 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330669 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330674 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330678 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330682 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:20.335756 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330688 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330692 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330696 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330700 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330704 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330709 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330713 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330717 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330722 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330726 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330730 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330734 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330739 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330743 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330747 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330752 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330756 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330761 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330765 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330770 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:20.336204 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330774 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330778 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330782 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330786 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330790 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330795 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330800 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330804 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330808 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330812 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330816 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330821 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330825 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330830 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330834 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330840 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330847 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330852 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330856 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:20.336794 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330861 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330865 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330869 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330873 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330877 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330882 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330886 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330890 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330896 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330902 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330907 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330911 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330916 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330920 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330924 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330928 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330933 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330936 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330941 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330945 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:20.337242 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330949 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330954 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330958 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330962 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330966 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330970 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330974 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330979 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330984 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330988 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330992 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.330996 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.331000 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.331004 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.331008 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.331012 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.331017 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.331023 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.331027 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.331031 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:20.337763 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.331037 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.331041 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.331045 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332004 2562 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332020 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332032 2562 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332040 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332047 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332052 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332059 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332066 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332071 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332077 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332082 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332088 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332093 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332098 2562 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332103 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332108 2562 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332112 2562 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332119 2562 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332126 2562 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332131 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332136 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:46:20.338230 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332141 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332146 2562 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332153 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332158 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332163 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332168 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332173 2562 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332177 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332183 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332188 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332198 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332208 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332213 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332219 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332224 2562 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332228 2562 flags.go:64] FLAG: --enable-server="true" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332233 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332241 2562 flags.go:64] FLAG: --event-burst="100" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332246 2562 flags.go:64] FLAG: --event-qps="50" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332251 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332256 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332261 2562 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332268 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332274 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332279 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:46:20.338817 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332285 2562 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332290 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332294 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332300 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332305 2562 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332310 2562 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332315 2562 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332320 2562 flags.go:64] FLAG: --feature-gates="" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332327 2562 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332332 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332337 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332342 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332347 2562 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332367 2562 flags.go:64] FLAG: --help="false" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332372 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332377 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332383 2562 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332391 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332398 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332404 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332408 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332413 2562 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332418 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:46:20.339641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332422 2562 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332427 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332432 2562 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332437 2562 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332442 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332446 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332451 2562 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332456 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332461 2562 flags.go:64] FLAG: --lock-file="" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332465 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332470 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332476 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332485 2562 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332489 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332496 2562 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332502 2562 flags.go:64] FLAG: --logging-format="text" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332507 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332512 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332517 2562 flags.go:64] FLAG: --manifest-url="" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332522 2562 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332529 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332533 2562 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332540 2562 flags.go:64] FLAG: --max-pods="110" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332545 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332551 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:46:20.340190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332556 2562 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332562 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332568 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332573 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332578 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332589 2562 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332595 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332600 2562 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332605 2562 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332610 2562 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332614 2562 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332619 2562 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332624 2562 flags.go:64] FLAG: --port="10250" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332629 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332634 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0404dcdd83f276fec" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332639 2562 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332644 2562 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332649 2562 flags.go:64] FLAG: --register-node="true" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332654 2562 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332660 2562 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332665 2562 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332670 2562 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332683 2562 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332690 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332695 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332701 2562 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:46:20.340795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332706 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332710 2562 flags.go:64] FLAG: --runonce="false" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332716 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332721 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332726 2562 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332730 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332735 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332742 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332750 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332755 2562 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332760 2562 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332764 2562 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332769 2562 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332774 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332779 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332784 2562 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332789 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332798 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332803 2562 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332808 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332814 2562 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332819 2562 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332823 2562 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332828 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332834 2562 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:46:20.341425 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332838 2562 flags.go:64] FLAG: --v="2" Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332846 2562 flags.go:64] FLAG: --version="false" Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332852 2562 flags.go:64] FLAG: --vmodule="" Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332859 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.332865 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333085 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333093 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333098 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333102 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333107 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333111 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333115 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333119 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333124 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333131 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333137 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333141 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333145 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333150 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333154 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333158 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:20.342022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333163 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333167 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333171 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333176 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333180 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333184 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333189 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333193 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333197 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333201 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333208 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333214 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333219 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333224 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333228 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333232 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333237 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333241 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333246 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:20.342582 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333250 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333254 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333258 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333263 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333267 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333272 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333279 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333284 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333288 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333293 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333297 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333302 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333306 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333310 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333315 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333319 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333323 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333327 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333332 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:20.343022 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333336 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333340 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333344 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333348 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333368 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333373 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333380 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333386 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333390 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333394 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333398 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333403 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333408 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333412 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333416 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333420 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333425 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333429 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333433 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:20.343591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333439 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333445 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333449 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333454 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333458 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333463 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333467 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333471 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333476 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333480 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333485 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333490 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333494 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333498 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333502 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333507 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333511 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333515 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333520 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333524 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:20.344044 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333528 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333533 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333537 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333541 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333545 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333551 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333555 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333560 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333563 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333568 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333572 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333577 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333583 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.333627 2562 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333684 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333693 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:20.344536 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333698 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333703 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333708 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333712 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333716 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333720 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333724 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333729 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333733 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333737 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333742 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333746 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333750 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333754 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333758 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333763 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333767 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333771 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333776 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333780 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:20.344933 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333785 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333789 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333794 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333798 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333802 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333807 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333811 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333815 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333821 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333825 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333831 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333835 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333839 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333843 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333848 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333853 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333857 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333861 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333865 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:20.345430 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333870 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333874 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333878 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333882 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333886 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333890 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333894 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333898 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333904 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333908 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333912 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333916 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333921 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333925 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333929 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333933 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333942 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333947 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333951 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333955 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:20.345883 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333959 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333965 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333970 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333974 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333978 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333982 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333986 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333991 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333995 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.333999 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334005 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334012 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334017 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334021 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334026 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334030 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334034 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334038 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334042 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:20.346371 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334046 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334051 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334055 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334059 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334064 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334068 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334072 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334076 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334080 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334084 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334089 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334093 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334098 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334102 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334108 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334112 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334116 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334121 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334125 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:20.346836 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334129 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334133 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334137 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334142 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334146 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334150 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.334158 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.341591 2562 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.341608 2562 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.341622 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.341626 2562 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341689 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341695 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341699 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341702 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341705 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341707 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341711 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341714 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:20.347297 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341717 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341720 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341723 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341726 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341729 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341732 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341734 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341737 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341739 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341742 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341745 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341747 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341750 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341752 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341755 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341758 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341761 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341763 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341766 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341768 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:20.347839 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341772 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341780 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341784 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341787 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341790 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341792 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341795 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341798 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341800 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341803 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341806 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341808 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341811 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341814 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341816 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341819 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341822 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341825 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341828 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:20.348337 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341830 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341833 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341836 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341839 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341841 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341844 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341847 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341849 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341852 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341855 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341857 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341860 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341862 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341867 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341871 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341880 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341883 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341886 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:20.349029 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341888 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341892 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341896 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341899 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341902 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341905 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341908 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341911 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341913 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341916 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341918 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341921 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341924 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341926 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341929 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341931 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341934 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341936 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341939 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341941 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:20.349722 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341944 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341947 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341949 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341952 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341954 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341957 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341959 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341963 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341966 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341968 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341976 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341979 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341982 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341985 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341987 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341990 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341992 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341995 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.341998 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342000 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:20.350269 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342003 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.342031 2562 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342223 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342229 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342232 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342235 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342238 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342241 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342244 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342247 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342249 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342252 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342254 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342257 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342260 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342262 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:20.350766 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342265 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342268 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342270 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342273 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342276 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342279 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342281 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342291 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342294 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342297 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342300 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342305 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342308 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342311 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342313 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342317 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342320 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342322 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342325 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:20.351133 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342327 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342330 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342334 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342337 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342340 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342343 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342346 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342348 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342368 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342373 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342376 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342379 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342382 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342385 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342387 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342390 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342393 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342396 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342399 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:20.351620 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342401 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342404 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342413 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342415 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342418 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342420 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342423 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342426 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342428 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342431 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342434 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342436 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342439 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342442 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342444 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342447 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342449 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342452 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342454 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:20.352107 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342457 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342460 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342463 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342465 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342468 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342471 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342473 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342476 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342478 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342481 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342484 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342486 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342489 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342492 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342494 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342497 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342505 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342508 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342511 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342513 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:20.352639 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342516 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342518 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342521 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342524 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342527 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342529 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342532 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342535 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342537 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342540 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342542 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342545 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342548 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342567 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342571 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:20.353110 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.342596 2562 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342634 2562 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342638 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342641 2562 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342644 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342647 2562 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342649 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342652 2562 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342654 2562 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342657 2562 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342666 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342669 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342672 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342675 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342683 2562 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342686 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342689 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342691 2562 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342694 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:46:20.353492 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342697 2562 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342700 2562 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342703 2562 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342706 2562 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342708 2562 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342711 2562 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342713 2562 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342716 2562 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342718 2562 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342721 2562 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342723 2562 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342726 2562 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342729 2562 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342732 2562 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342734 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342736 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342739 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342742 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342744 2562 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342747 2562 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:46:20.353938 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342750 2562 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342752 2562 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342755 2562 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342758 2562 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342761 2562 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342763 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342766 2562 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342769 2562 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342778 2562 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342781 2562 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342783 2562 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342786 2562 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342790 2562 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342794 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342797 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342799 2562 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342802 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342805 2562 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342809 2562 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:46:20.354439 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342812 2562 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342815 2562 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342819 2562 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342821 2562 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342824 2562 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342826 2562 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342829 2562 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342831 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342834 2562 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342837 2562 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342839 2562 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342842 2562 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342845 2562 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342848 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342850 2562 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342853 2562 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342858 2562 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342861 2562 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342863 2562 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342866 2562 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:46:20.354880 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342869 2562 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342871 2562 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342879 2562 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342882 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342885 2562 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342887 2562 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342890 2562 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342892 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342895 2562 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342898 2562 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342900 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342903 2562 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342905 2562 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342908 2562 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342911 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342913 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342915 2562 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342918 2562 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342921 2562 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:46:20.355404 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342923 2562 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342926 2562 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342928 2562 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342931 2562 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342933 2562 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342936 2562 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342939 2562 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342942 2562 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342945 2562 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:20.342948 2562 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.343024 2562 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.345176 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.346438 2562 server.go:1014] "Starting client certificate rotation" Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.346544 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.347270 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.351912 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:46:20.355852 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.355732 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:46:20.365659 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.365641 2562 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:46:20.368733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.368716 2562 log.go:25] "Validated CRI v1 image API" Oct 30 16:46:20.370116 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.370090 2562 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:46:20.373891 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.373871 2562 fs.go:135] Filesystem UUIDs: map[7144d48a-3974-4cc6-9382-77cb4226ad18:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b137222b-6488-4ff5-8450-70be74d5ba21:/dev/nvme0n1p4] Oct 30 16:46:20.373966 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.373890 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 16:46:20.379834 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.379723 2562 manager.go:217] Machine: {Timestamp:2026-10-30 16:46:20.378470679 +0000 UTC m=+0.315218489 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100226 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20c9094790d091b2a7c2bb0d4f7aac SystemUUID:ec20c909-4790-d091-b2a7-c2bb0d4f7aac BootID:8b035d1e-6263-40d4-b37d-0da56a929adb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d9:80:b5:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d9:80:b5:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:13:eb:24:83:01 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:46:20.380463 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.380451 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:46:20.380591 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.380578 2562 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:46:20.383108 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.383085 2562 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:46:20.383263 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.383109 2562 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-153.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:46:20.383308 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.383266 2562 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:46:20.383308 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.383274 2562 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:46:20.383308 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.383300 2562 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:46:20.383398 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.383317 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:46:20.384281 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.384271 2562 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:46:20.387948 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.387935 2562 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:46:20.388069 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.388060 2562 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:46:20.388658 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.388649 2562 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:46:20.388689 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.388664 2562 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:46:20.388689 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.388677 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:46:20.388689 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.388686 2562 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:46:20.388816 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.388693 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:46:20.389152 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.389137 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:46:20.391338 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.391312 2562 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:46:20.396733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.396716 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:46:20.397451 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.397436 2562 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:46:20.397513 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.397467 2562 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:46:20.397624 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.397614 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:46:20.397655 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.397630 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:46:20.397655 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.397637 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:46:20.397655 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.397643 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:46:20.397655 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.397649 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:46:20.397655 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.397655 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:46:20.398104 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.398095 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:46:20.398135 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.398104 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:46:20.398135 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.398112 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:46:20.398135 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.398119 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:46:20.398135 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.398128 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:46:20.398135 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.398136 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:46:20.398673 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.398662 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:46:20.398673 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.398673 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:46:20.402365 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.402344 2562 server.go:1263] "Started kubelet" Oct 30 16:46:20.402453 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.402423 2562 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:46:20.402548 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.402473 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:46:20.402584 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.402572 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:46:20.403139 ip-10-0-135-153 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:46:20.403858 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.403835 2562 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:46:20.405023 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.405009 2562 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:46:20.408084 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.408061 2562 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 16:46:20.412154 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.412132 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:46:20.412545 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.412529 2562 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:46:20.412631 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.412601 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:20.413093 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.413069 2562 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:46:20.413093 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.413092 2562 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:46:20.413217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.413067 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:46:20.413267 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.413245 2562 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:46:20.413267 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.413254 2562 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:46:20.413469 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.413440 2562 factory.go:55] Registering systemd factory Oct 30 16:46:20.413469 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.413466 2562 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:46:20.413715 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.413702 2562 factory.go:153] Registering CRI-O factory Oct 30 16:46:20.413755 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.413716 2562 factory.go:223] Registration of the crio container factory successfully Oct 30 16:46:20.413784 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.413757 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:46:20.413784 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.413771 2562 factory.go:103] Registering Raw factory Oct 30 16:46:20.413784 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.413782 2562 manager.go:1199] Started watching for new ooms in manager Oct 30 16:46:20.414589 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.414566 2562 manager.go:319] Starting recovery of all containers Oct 30 16:46:20.422020 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.421976 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:46:20.424961 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.424939 2562 manager.go:324] Recovery completed Oct 30 16:46:20.431936 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.431920 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:20.433581 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.433562 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:20.433665 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.433597 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:20.433665 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.433613 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:20.434176 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.434164 2562 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:46:20.434176 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.434175 2562 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:46:20.434176 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.434182 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:20.434349 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.434194 2562 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:46:20.434830 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.434813 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:20.434903 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.434839 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:20.434903 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.434848 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:20.436383 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.436372 2562 policy_none.go:50] "Start" Oct 30 16:46:20.436427 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.436387 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:46:20.436427 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.436397 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:46:20.438471 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.438454 2562 policy_none.go:44] "Start" Oct 30 16:46:20.440683 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.440660 2562 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-153.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 16:46:20.440683 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.440664 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-153.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:46:20.443350 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.440659 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-153.ec2.internal.18e35c7fc19a75ef default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-153.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-153.ec2.internal,},FirstTimestamp:2026-10-30 16:46:20.402316783 +0000 UTC m=+0.339064595,LastTimestamp:2026-10-30 16:46:20.402316783 +0000 UTC m=+0.339064595,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-153.ec2.internal,}" Oct 30 16:46:20.447416 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.447400 2562 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:46:20.457132 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.457091 2562 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 16:46:20.457132 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.457091 2562 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 16:46:20.470064 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.470021 2562 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:46:20.472781 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.472707 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-153.ec2.internal.18e35c7fc3777e35 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-153.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-153.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-153.ec2.internal,},FirstTimestamp:2026-10-30 16:46:20.433579573 +0000 UTC m=+0.370327389,LastTimestamp:2026-10-30 16:46:20.433579573 +0000 UTC m=+0.370327389,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-153.ec2.internal,}" Oct 30 16:46:20.473013 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.472984 2562 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-153.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 16:46:20.474778 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.474764 2562 manager.go:341] "Starting Device Plugin manager" Oct 30 16:46:20.474896 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.474879 2562 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:46:20.474957 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.474897 2562 server.go:87] "Starting device plugin registration server" Oct 30 16:46:20.475219 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.475205 2562 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:46:20.475275 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.475221 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:46:20.475381 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.475332 2562 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:46:20.475466 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.475452 2562 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:46:20.475514 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.475468 2562 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:46:20.475997 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.475926 2562 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:46:20.475997 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.475977 2562 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:20.489500 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.489417 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-153.ec2.internal.18e35c7fc377dea1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-153.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-153.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-153.ec2.internal,},FirstTimestamp:2026-10-30 16:46:20.433604257 +0000 UTC m=+0.370352070,LastTimestamp:2026-10-30 16:46:20.433604257 +0000 UTC m=+0.370352070,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-153.ec2.internal,}" Oct 30 16:46:20.514305 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.497462 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ptdh2" Oct 30 16:46:20.514305 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.505958 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-153.ec2.internal.18e35c7fc37819ce default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-153.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-153.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-153.ec2.internal,},FirstTimestamp:2026-10-30 16:46:20.433619406 +0000 UTC m=+0.370367222,LastTimestamp:2026-10-30 16:46:20.433619406 +0000 UTC m=+0.370367222,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-153.ec2.internal,}" Oct 30 16:46:20.514305 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.513407 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ptdh2" Oct 30 16:46:20.526444 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.526351 2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-153.ec2.internal.18e35c7fc3777e35\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-153.ec2.internal.18e35c7fc3777e35 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-153.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-153.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-153.ec2.internal,},FirstTimestamp:2026-10-30 16:46:20.433579573 +0000 UTC m=+0.370327389,LastTimestamp:2026-10-30 16:46:20.434827078 +0000 UTC m=+0.371574899,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-153.ec2.internal,}" Oct 30 16:46:20.542268 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.542236 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:46:20.542268 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.542267 2562 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:46:20.542451 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.542288 2562 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:46:20.542451 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.542321 2562 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 16:46:20.559878 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.559854 2562 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:46:20.575757 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.575703 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:20.576995 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.576977 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:20.577069 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.577009 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:20.577069 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.577020 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:20.577069 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.577045 2562 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.599953 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.599928 2562 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.600049 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.599957 2562 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-153.ec2.internal\": node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:20.643123 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.643097 2562 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal"] Oct 30 16:46:20.643228 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.643155 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:20.644089 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.644073 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:20.644146 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.644105 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:20.644146 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.644116 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:20.646391 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.646379 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:20.646532 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.646518 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.646562 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.646549 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:20.655840 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.655819 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:20.655933 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.655851 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:20.655933 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.655827 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:20.655933 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.655880 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:20.655933 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.655912 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:20.655933 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.655930 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:20.658061 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.658047 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.658140 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.658071 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:20.658953 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.658935 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:20.659024 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.658963 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:20.659024 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.658976 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:20.669036 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.669018 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:20.677021 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.677004 2562 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-153.ec2.internal\" not found" node="ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.679101 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.679083 2562 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-153.ec2.internal\" not found" node="ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.769430 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.769400 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:20.814786 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.814754 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9b4913d9c6bf71319283ff1eade5a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" (UID: \"b9b4913d9c6bf71319283ff1eade5a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.814901 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.814791 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9b4913d9c6bf71319283ff1eade5a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" (UID: \"b9b4913d9c6bf71319283ff1eade5a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.814901 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.814809 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8216a0ebd9e778d471b8e472a27d4665-config\") pod \"kube-apiserver-proxy-ip-10-0-135-153.ec2.internal\" (UID: \"8216a0ebd9e778d471b8e472a27d4665\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.870345 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.870281 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:20.915609 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.915577 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9b4913d9c6bf71319283ff1eade5a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" (UID: \"b9b4913d9c6bf71319283ff1eade5a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.915753 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.915612 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9b4913d9c6bf71319283ff1eade5a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" (UID: \"b9b4913d9c6bf71319283ff1eade5a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.915753 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.915648 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8216a0ebd9e778d471b8e472a27d4665-config\") pod \"kube-apiserver-proxy-ip-10-0-135-153.ec2.internal\" (UID: \"8216a0ebd9e778d471b8e472a27d4665\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.915753 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.915673 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9b4913d9c6bf71319283ff1eade5a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" (UID: \"b9b4913d9c6bf71319283ff1eade5a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.915753 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.915685 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8216a0ebd9e778d471b8e472a27d4665-config\") pod \"kube-apiserver-proxy-ip-10-0-135-153.ec2.internal\" (UID: \"8216a0ebd9e778d471b8e472a27d4665\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.915753 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.915686 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9b4913d9c6bf71319283ff1eade5a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" (UID: \"b9b4913d9c6bf71319283ff1eade5a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.971035 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:20.971005 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:20.980206 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.980182 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:46:20.983326 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:20.983309 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" Oct 30 16:46:21.072119 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:21.071933 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:21.172467 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:21.172393 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:21.272891 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:21.272857 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:22.600996 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:22.600953 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 16:46:22.601446 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:22.601113 2562 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 16:46:22.627555 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:22.627524 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:22.690573 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:22.690504 2562 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:46:22.728562 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:22.728534 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:22.772694 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:22.772657 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:41:21 +0000 UTC" deadline="2028-07-27 05:52:53.28376147 +0000 UTC" Oct 30 16:46:22.772694 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:22.772689 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15253h6m30.511076302s" Oct 30 16:46:22.825962 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:22.825935 2562 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:46:22.826852 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:22.826831 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:46:22.826960 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:22.826866 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:46:22.826960 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:22.826878 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:46:22.829066 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:22.829047 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:22.847082 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:22.847045 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b4913d9c6bf71319283ff1eade5a9e.slice/crio-3d9a4c93cd8726956f1a8f7901706b5a824a9ce074b05b8adb54875dd8ecd2bd WatchSource:0}: Error finding container 3d9a4c93cd8726956f1a8f7901706b5a824a9ce074b05b8adb54875dd8ecd2bd: Status 404 returned error can't find the container with id 3d9a4c93cd8726956f1a8f7901706b5a824a9ce074b05b8adb54875dd8ecd2bd Oct 30 16:46:22.854486 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:22.854471 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:46:22.907943 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:22.907914 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8216a0ebd9e778d471b8e472a27d4665.slice/crio-d7120dbf5c69333bd3e5babceeda19cf41131e73908d132cd7c31a3b7b882137 WatchSource:0}: Error finding container d7120dbf5c69333bd3e5babceeda19cf41131e73908d132cd7c31a3b7b882137: Status 404 returned error can't find the container with id d7120dbf5c69333bd3e5babceeda19cf41131e73908d132cd7c31a3b7b882137 Oct 30 16:46:22.929861 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:22.929841 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:22.957019 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:22.956966 2562 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:46:23.030498 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.030454 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:23.131176 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.131148 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:23.231893 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.231819 2562 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:46:23.328477 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.328444 2562 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:46:23.367522 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.367483 2562 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:46:23.394642 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.394616 2562 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:46:23.395845 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.395817 2562 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" Oct 30 16:46:23.417743 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.417633 2562 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:46:23.645355 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.645285 2562 apiserver.go:52] "Watching apiserver" Oct 30 16:46:23.668428 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.668397 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 16:46:23.688936 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.688893 2562 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:46:23.689283 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.689259 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-nbbqf","openshift-multus/multus-additional-cni-plugins-2hh98","openshift-multus/multus-q9f8r","openshift-network-diagnostics/network-check-target-scdcl","openshift-image-registry/node-ca-hfsmw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-2ljft","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl","openshift-multus/network-metrics-daemon-8mpgm","openshift-network-operator/iptables-alerter-l4rsf","kube-system/global-pull-secret-syncer-wgxv6","kube-system/konnectivity-agent-4hlmr","kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal","openshift-cluster-node-tuning-operator/tuned-c7ckl"] Oct 30 16:46:23.691796 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.691764 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:46:23.693909 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.693874 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.697020 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.697001 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:46:23.699390 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.699368 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.701616 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.701592 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.705450 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.705430 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:46:23.707588 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.707567 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:23.707678 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.707634 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:23.710158 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.710134 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:46:23.710158 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.710153 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:46:23.710299 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.710185 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:46:23.710299 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.710195 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:46:23.710299 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.710251 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:46:23.710299 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.710271 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rbwcw\"" Oct 30 16:46:23.710299 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.710280 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fxkbf\"" Oct 30 16:46:23.711544 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.711528 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:23.711641 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.711584 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:23.713552 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.713524 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:46:23.713641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.713565 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z7xnd\"" Oct 30 16:46:23.715814 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.715789 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:23.715892 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.715848 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:23.718345 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.718324 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.720661 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.720637 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.723028 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.723003 2562 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:46:23.728883 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.728864 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dv9t8" Oct 30 16:46:23.740104 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.739663 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dv9t8" Oct 30 16:46:23.768715 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.768694 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:46:23.778221 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778187 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72lmm\" (UniqueName: \"kubernetes.io/projected/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-kube-api-access-72lmm\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:46:23.778355 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-registration-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.778355 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778256 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-systemd-units\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.778355 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-cni-netd\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.778355 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778305 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-conf-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.778355 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778327 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-multus-certs\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.778355 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778351 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmzz5\" (UniqueName: \"kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5\") pod \"network-check-target-scdcl\" (UID: \"a132686f-9475-437d-826e-22fb93e86323\") " pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:23.778630 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778375 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-tuned\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.778630 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778400 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-system-cni-dir\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.778630 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778422 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-host\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:46:23.778630 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778444 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/852f74bb-c91e-4075-8335-aeba3e723709-ovn-node-metrics-cert\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.778630 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778468 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-lib-modules\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.778630 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778489 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nttq\" (UniqueName: \"kubernetes.io/projected/8c30c30a-8c1f-4552-b2d1-2692bc423d45-kube-api-access-9nttq\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.778630 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-netns\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.778630 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778535 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-device-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.778630 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-774sr\" (UniqueName: \"kubernetes.io/projected/252c7435-f4c1-4dac-a468-e3287c6a7e9e-kube-api-access-774sr\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.778630 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778600 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-slash\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.778630 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-log-socket\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.779125 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778665 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-socket-dir-parent\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.779125 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778692 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-etc-kubernetes\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.779125 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-systemd\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.779125 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778802 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-cni-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.779125 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778835 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvsrx\" (UniqueName: \"kubernetes.io/projected/a41c35af-f159-4802-9b53-fbb8cbee2104-kube-api-access-rvsrx\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.779125 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778889 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-node-log\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.779125 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778937 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-run-ovn-kubernetes\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.779125 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.778979 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp752\" (UniqueName: \"kubernetes.io/projected/852f74bb-c91e-4075-8335-aeba3e723709-kube-api-access-vp752\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.779125 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779024 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-dbus\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:23.779125 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779054 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-system-cni-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.779125 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779094 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-daemon-config\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779120 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-run-netns\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779156 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779180 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-ovnkube-config\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779223 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-modprobe-d\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-kubernetes\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779262 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysctl-conf\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779278 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-host\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779293 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c30c30a-8c1f-4552-b2d1-2692bc423d45-tmp\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779308 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-cnibin\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779352 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-env-overrides\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779415 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b174c80c-af6a-4f17-8442-1a029b286011-host-slash\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779441 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l2kn\" (UniqueName: \"kubernetes.io/projected/b54c34b7-e8c3-4fe9-9b66-36336dfae405-kube-api-access-4l2kn\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779464 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-tmp-dir\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:46:23.779491 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779491 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:23.780088 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779516 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-hostroot\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.780088 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779545 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-run\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.780088 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.780088 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779595 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:23.780088 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779619 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-systemd\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.780088 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779642 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a-agent-certs\") pod \"konnectivity-agent-4hlmr\" (UID: \"9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a\") " pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:46:23.780088 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779693 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a-konnectivity-ca\") pod \"konnectivity-agent-4hlmr\" (UID: \"9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a\") " pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:46:23.780088 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779727 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.780088 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779765 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-ovnkube-script-lib\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.780088 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779809 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-kubelet-config\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:23.780088 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.779847 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-cnibin\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780179 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-cni-bin\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780207 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-socket-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780255 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-hosts-file\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780269 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svsvb\" (UniqueName: \"kubernetes.io/projected/3ca4c236-e28d-4331-9336-63a71d706ef0-kube-api-access-svsvb\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780282 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-k8s-cni-cncf-io\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780295 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-cni-multus\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780309 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysctl-d\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780324 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780338 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-var-lib-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780352 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-etc-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780367 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b54c34b7-e8c3-4fe9-9b66-36336dfae405-cni-binary-copy\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780380 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysconfig\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780398 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-var-lib-kubelet\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780418 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-serviceca\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:46:23.780481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780436 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.781026 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780449 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-kubelet\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.781026 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780462 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-sys-fs\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.781026 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780475 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-kubelet\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.781026 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780489 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-ovn\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.781026 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b174c80c-af6a-4f17-8442-1a029b286011-iptables-alerter-script\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:46:23.781026 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780517 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w2qd\" (UniqueName: \"kubernetes.io/projected/b174c80c-af6a-4f17-8442-1a029b286011-kube-api-access-6w2qd\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:46:23.781026 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780531 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-sys\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.781026 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780555 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-cni-binary-copy\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.781026 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwhw\" (UniqueName: \"kubernetes.io/projected/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-kube-api-access-rvwhw\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:46:23.781026 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.780598 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-cni-bin\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.802101 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.802051 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" event={"ID":"8216a0ebd9e778d471b8e472a27d4665","Type":"ContainerStarted","Data":"d7120dbf5c69333bd3e5babceeda19cf41131e73908d132cd7c31a3b7b882137"} Oct 30 16:46:23.803066 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.803043 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" event={"ID":"b9b4913d9c6bf71319283ff1eade5a9e","Type":"ContainerStarted","Data":"3d9a4c93cd8726956f1a8f7901706b5a824a9ce074b05b8adb54875dd8ecd2bd"} Oct 30 16:46:23.808932 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.808703 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:46:23.808932 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.808721 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:46:23.808932 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.808739 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:46:23.808932 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.808763 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:46:23.808932 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.808791 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:46:23.808932 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.808793 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:46:23.808932 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.808844 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:46:23.808932 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.808855 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:46:23.808932 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.808866 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:46:23.809380 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809093 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:46:23.809380 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809158 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:46:23.809380 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809263 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:46:23.809380 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809282 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:46:23.809380 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809326 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:46:23.809380 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809331 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:46:23.809649 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809378 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wd4nn\"" Oct 30 16:46:23.809649 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809427 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:46:23.809649 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809505 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pdblj\"" Oct 30 16:46:23.809649 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809596 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7c68\"" Oct 30 16:46:23.809649 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809604 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:46:23.809649 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809615 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:46:23.809649 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809619 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:46:23.809895 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809774 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8fhb8\"" Oct 30 16:46:23.809895 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809841 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:46:23.809895 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.809869 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j4x52\"" Oct 30 16:46:23.820288 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.820262 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:46:23.820288 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.820263 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:46:23.820468 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.820338 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6bljg\"" Oct 30 16:46:23.881657 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.881621 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-sys-fs\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.881657 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.881663 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-kubelet\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.881918 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.881718 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-ovn\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.881918 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.881741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b174c80c-af6a-4f17-8442-1a029b286011-iptables-alerter-script\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:46:23.881918 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.881768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2qd\" (UniqueName: \"kubernetes.io/projected/b174c80c-af6a-4f17-8442-1a029b286011-kube-api-access-6w2qd\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:46:23.881918 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.881791 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-sys\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.881918 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.881856 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-cni-binary-copy\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.881918 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.881888 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwhw\" (UniqueName: \"kubernetes.io/projected/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-kube-api-access-rvwhw\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:46:23.882217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.881929 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-cni-bin\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.882217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.881950 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72lmm\" (UniqueName: \"kubernetes.io/projected/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-kube-api-access-72lmm\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:46:23.882217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.881974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-registration-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.882217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-systemd-units\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.882217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882045 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-cni-netd\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.882217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882066 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-sys-fs\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.882217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882070 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-conf-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.882217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882076 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-kubelet\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.882217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882109 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-multus-certs\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.882217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzz5\" (UniqueName: \"kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5\") pod \"network-check-target-scdcl\" (UID: \"a132686f-9475-437d-826e-22fb93e86323\") " pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:23.882217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882179 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-tuned\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.882217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-system-cni-dir\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.882217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882214 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-registration-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882226 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-host\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882255 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-host\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/852f74bb-c91e-4075-8335-aeba3e723709-ovn-node-metrics-cert\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882285 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-multus-certs\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-lib-modules\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882346 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nttq\" (UniqueName: \"kubernetes.io/projected/8c30c30a-8c1f-4552-b2d1-2692bc423d45-kube-api-access-9nttq\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882373 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-netns\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882399 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-device-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882423 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-774sr\" (UniqueName: \"kubernetes.io/projected/252c7435-f4c1-4dac-a468-e3287c6a7e9e-kube-api-access-774sr\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882488 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-slash\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-log-socket\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-socket-dir-parent\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-etc-kubernetes\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.882641 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882587 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-systemd\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882664 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-sys\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882667 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-cni-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882717 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvsrx\" (UniqueName: \"kubernetes.io/projected/a41c35af-f159-4802-9b53-fbb8cbee2104-kube-api-access-rvsrx\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882753 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-node-log\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882776 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-run-ovn-kubernetes\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882812 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp752\" (UniqueName: \"kubernetes.io/projected/852f74bb-c91e-4075-8335-aeba3e723709-kube-api-access-vp752\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882835 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-dbus\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-system-cni-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882937 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-daemon-config\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882954 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-system-cni-dir\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883017 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-run-netns\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882961 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-run-netns\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883102 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-log-socket\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883102 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883145 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883139 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-node-log\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883158 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-systemd-units\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883195 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-cni-netd\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882044 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-ovn\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883236 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-conf-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883262 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-socket-dir-parent\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883303 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-cni-binary-copy\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883333 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-etc-kubernetes\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883394 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-cni-bin\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883447 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-systemd\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883497 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-cni-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883541 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b174c80c-af6a-4f17-8442-1a029b286011-iptables-alerter-script\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883564 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-run-ovn-kubernetes\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883598 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-lib-modules\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883598 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-device-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883630 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-netns\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883512 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-dbus\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883653 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-system-cni-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.883696 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.882877 2562 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883780 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-slash\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883809 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-ovnkube-config\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883856 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-modprobe-d\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-kubernetes\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883947 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysctl-conf\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883970 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-host\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.883991 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c30c30a-8c1f-4552-b2d1-2692bc423d45-tmp\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884012 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-cnibin\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884107 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884133 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-env-overrides\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b174c80c-af6a-4f17-8442-1a029b286011-host-slash\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l2kn\" (UniqueName: \"kubernetes.io/projected/b54c34b7-e8c3-4fe9-9b66-36336dfae405-kube-api-access-4l2kn\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-tmp-dir\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884242 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-daemon-config\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884251 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884279 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-hostroot\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.884420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884296 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-run\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884313 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884334 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884349 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-systemd\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a-agent-certs\") pod \"konnectivity-agent-4hlmr\" (UID: \"9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a\") " pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.884382 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a-konnectivity-ca\") pod \"konnectivity-agent-4hlmr\" (UID: \"9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a\") " pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884449 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-ovnkube-config\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884456 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.884459 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret podName:ac8ed89d-9808-4bff-9d24-3e5fc34041ad nodeName:}" failed. No retries permitted until 2026-10-30 16:46:24.384427521 +0000 UTC m=+3.066630702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret") pod "global-pull-secret-syncer-wgxv6" (UID: "ac8ed89d-9808-4bff-9d24-3e5fc34041ad") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-ovnkube-script-lib\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.884566 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.884610 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs podName:3ca4c236-e28d-4331-9336-63a71d706ef0 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:24.384598058 +0000 UTC m=+3.066801220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs") pod "network-metrics-daemon-8mpgm" (UID: "3ca4c236-e28d-4331-9336-63a71d706ef0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884632 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-cnibin\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884668 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-hostroot\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884719 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-modprobe-d\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.885121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884766 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-kubelet-config\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:23.885861 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884638 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-kubelet-config\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:23.885861 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.884839 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-run\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.885861 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.885177 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.885861 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.885256 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-env-overrides\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.885861 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.885508 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-tmp-dir\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:46:23.885861 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.885577 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-kubernetes\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.885861 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.885657 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a-konnectivity-ca\") pod \"konnectivity-agent-4hlmr\" (UID: \"9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a\") " pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:46:23.885861 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.885682 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysctl-conf\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.885861 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.885715 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b174c80c-af6a-4f17-8442-1a029b286011-host-slash\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:46:23.885861 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.885731 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-host\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.886345 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886022 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-cnibin\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.886345 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886052 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-cni-bin\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.886345 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886054 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-systemd\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.886345 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886106 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.886345 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-socket-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.886345 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886190 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.886345 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886216 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-hosts-file\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:46:23.886345 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886290 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svsvb\" (UniqueName: \"kubernetes.io/projected/3ca4c236-e28d-4331-9336-63a71d706ef0-kube-api-access-svsvb\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:23.886345 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886327 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-k8s-cni-cncf-io\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.886718 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886351 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-cni-multus\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.886718 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysctl-d\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.886718 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886387 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.886718 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-var-lib-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.886718 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886422 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-etc-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.886718 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b54c34b7-e8c3-4fe9-9b66-36336dfae405-cni-binary-copy\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.886718 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886517 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysconfig\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.886718 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886540 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-var-lib-kubelet\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.886718 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-serviceca\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:46:23.886718 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.886718 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886626 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-kubelet\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.886718 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886686 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-kubelet\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.886718 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886709 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-etc-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.887332 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886818 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-socket-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.887332 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886835 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.887332 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886880 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-cnibin\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.887332 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886939 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-ovnkube-script-lib\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.887332 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886949 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-cni-bin\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.887332 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.886997 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-var-lib-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.887332 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.887041 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-hosts-file\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:46:23.887332 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.887094 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-k8s-cni-cncf-io\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.887332 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.887132 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-cni-multus\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.887332 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.887216 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysctl-d\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.887332 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.887232 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.887791 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.887420 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-serviceca\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:46:23.887791 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.887461 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-var-lib-kubelet\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.887791 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.887627 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysconfig\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.889925 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.888336 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.889925 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.888559 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b54c34b7-e8c3-4fe9-9b66-36336dfae405-cni-binary-copy\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.889925 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.888617 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-tuned\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.889925 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.888730 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/852f74bb-c91e-4075-8335-aeba3e723709-ovn-node-metrics-cert\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.891735 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.891713 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a-agent-certs\") pod \"konnectivity-agent-4hlmr\" (UID: \"9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a\") " pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:46:23.892424 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.892401 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c30c30a-8c1f-4552-b2d1-2692bc423d45-tmp\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.905509 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.905412 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:46:23.905509 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.905467 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:46:23.905509 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.905483 2562 projected.go:196] Error preparing data for projected volume kube-api-access-cmzz5 for pod openshift-network-diagnostics/network-check-target-scdcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:23.905778 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:23.905558 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5 podName:a132686f-9475-437d-826e-22fb93e86323 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:24.405536521 +0000 UTC m=+3.087739700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cmzz5" (UniqueName: "kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5") pod "network-check-target-scdcl" (UID: "a132686f-9475-437d-826e-22fb93e86323") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:23.908937 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.908812 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwhw\" (UniqueName: \"kubernetes.io/projected/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-kube-api-access-rvwhw\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:46:23.909849 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.909552 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72lmm\" (UniqueName: \"kubernetes.io/projected/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-kube-api-access-72lmm\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:46:23.909849 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.909809 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2qd\" (UniqueName: \"kubernetes.io/projected/b174c80c-af6a-4f17-8442-1a029b286011-kube-api-access-6w2qd\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:46:23.910825 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.910793 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nttq\" (UniqueName: \"kubernetes.io/projected/8c30c30a-8c1f-4552-b2d1-2692bc423d45-kube-api-access-9nttq\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:23.910932 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.910882 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-774sr\" (UniqueName: \"kubernetes.io/projected/252c7435-f4c1-4dac-a468-e3287c6a7e9e-kube-api-access-774sr\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:23.911288 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.911263 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvsrx\" (UniqueName: \"kubernetes.io/projected/a41c35af-f159-4802-9b53-fbb8cbee2104-kube-api-access-rvsrx\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:23.912288 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.912265 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svsvb\" (UniqueName: \"kubernetes.io/projected/3ca4c236-e28d-4331-9336-63a71d706ef0-kube-api-access-svsvb\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:23.913046 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.913026 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l2kn\" (UniqueName: \"kubernetes.io/projected/b54c34b7-e8c3-4fe9-9b66-36336dfae405-kube-api-access-4l2kn\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:46:23.913289 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.913266 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp752\" (UniqueName: \"kubernetes.io/projected/852f74bb-c91e-4075-8335-aeba3e723709-kube-api-access-vp752\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:23.978688 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:23.978648 2562 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:46:24.007153 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.007120 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:46:24.017562 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.017535 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:46:24.019114 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:24.019081 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b97c2d_82a0_47b2_aa06_c77c9c9ca67f.slice/crio-a7b568d421f6a612f9dd683fb4c69f0afba11581168228b603e39f73a03a9c6a WatchSource:0}: Error finding container a7b568d421f6a612f9dd683fb4c69f0afba11581168228b603e39f73a03a9c6a: Status 404 returned error can't find the container with id a7b568d421f6a612f9dd683fb4c69f0afba11581168228b603e39f73a03a9c6a Oct 30 16:46:24.026780 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:24.026741 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c30c30a_8c1f_4552_b2d1_2692bc423d45.slice/crio-69975af90d98b8a7f922c1e5a7d172dd00cd1588810fe804987d9a300241d3aa WatchSource:0}: Error finding container 69975af90d98b8a7f922c1e5a7d172dd00cd1588810fe804987d9a300241d3aa: Status 404 returned error can't find the container with id 69975af90d98b8a7f922c1e5a7d172dd00cd1588810fe804987d9a300241d3aa Oct 30 16:46:24.027993 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.027970 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:46:24.038803 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:24.038774 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0a1c6a_9bae_4e63_a98b_a5ff600ae63a.slice/crio-612ff75fe18ddb930ae5640e590053e6d754dd2a49f4c9f0a7259c66b8f97ff2 WatchSource:0}: Error finding container 612ff75fe18ddb930ae5640e590053e6d754dd2a49f4c9f0a7259c66b8f97ff2: Status 404 returned error can't find the container with id 612ff75fe18ddb930ae5640e590053e6d754dd2a49f4c9f0a7259c66b8f97ff2 Oct 30 16:46:24.039583 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.039561 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:46:24.049530 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:24.049505 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252c7435_f4c1_4dac_a468_e3287c6a7e9e.slice/crio-0cb0b06adaba74a0251d98b36fe0ad9508165fa9cc4a372a21cfcef74e6f9967 WatchSource:0}: Error finding container 0cb0b06adaba74a0251d98b36fe0ad9508165fa9cc4a372a21cfcef74e6f9967: Status 404 returned error can't find the container with id 0cb0b06adaba74a0251d98b36fe0ad9508165fa9cc4a372a21cfcef74e6f9967 Oct 30 16:46:24.057597 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.057576 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:46:24.062519 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.062494 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:24.064525 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.064508 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:46:24.068669 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.068650 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q9f8r" Oct 30 16:46:24.069529 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:24.069505 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8aeb454_5336_4fc1_8194_54d54a6cdb6c.slice/crio-2baff9f0fa3824f1aab8765d996070f22b46a05fe7497b4026d9d8798008ac70 WatchSource:0}: Error finding container 2baff9f0fa3824f1aab8765d996070f22b46a05fe7497b4026d9d8798008ac70: Status 404 returned error can't find the container with id 2baff9f0fa3824f1aab8765d996070f22b46a05fe7497b4026d9d8798008ac70 Oct 30 16:46:24.075591 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:24.075565 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852f74bb_c91e_4075_8335_aeba3e723709.slice/crio-035c8825d19d48aecd545e8171275583b3451316fc2fadf02d18814fa6f0a578 WatchSource:0}: Error finding container 035c8825d19d48aecd545e8171275583b3451316fc2fadf02d18814fa6f0a578: Status 404 returned error can't find the container with id 035c8825d19d48aecd545e8171275583b3451316fc2fadf02d18814fa6f0a578 Oct 30 16:46:24.079621 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:24.079595 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb174c80c_af6a_4f17_8442_1a029b286011.slice/crio-06bbf6f61b78f3ae3784d719972af3817a864582ffed7a1bd6f7e5b92d011189 WatchSource:0}: Error finding container 06bbf6f61b78f3ae3784d719972af3817a864582ffed7a1bd6f7e5b92d011189: Status 404 returned error can't find the container with id 06bbf6f61b78f3ae3784d719972af3817a864582ffed7a1bd6f7e5b92d011189 Oct 30 16:46:24.083296 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:24.083274 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54c34b7_e8c3_4fe9_9b66_36336dfae405.slice/crio-1be7b177c8467d07515635775555327fa59b33db1f3cf2100c5f2e6ef349493d WatchSource:0}: Error finding container 1be7b177c8467d07515635775555327fa59b33db1f3cf2100c5f2e6ef349493d: Status 404 returned error can't find the container with id 1be7b177c8467d07515635775555327fa59b33db1f3cf2100c5f2e6ef349493d Oct 30 16:46:24.089639 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.089617 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:46:24.101965 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:24.101935 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41c35af_f159_4802_9b53_fbb8cbee2104.slice/crio-d210d1881661386240975f06970c0bb1710c5358b3d15c73a1eb91b9f52dcbcf WatchSource:0}: Error finding container d210d1881661386240975f06970c0bb1710c5358b3d15c73a1eb91b9f52dcbcf: Status 404 returned error can't find the container with id d210d1881661386240975f06970c0bb1710c5358b3d15c73a1eb91b9f52dcbcf Oct 30 16:46:24.391235 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.391191 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:24.391390 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.391253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:24.391390 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:24.391334 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:24.391470 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:24.391402 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret podName:ac8ed89d-9808-4bff-9d24-3e5fc34041ad nodeName:}" failed. No retries permitted until 2026-10-30 16:46:25.391386455 +0000 UTC m=+4.073589631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret") pod "global-pull-secret-syncer-wgxv6" (UID: "ac8ed89d-9808-4bff-9d24-3e5fc34041ad") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:24.391470 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:24.391412 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:24.391570 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:24.391509 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs podName:3ca4c236-e28d-4331-9336-63a71d706ef0 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:25.391491077 +0000 UTC m=+4.073694251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs") pod "network-metrics-daemon-8mpgm" (UID: "3ca4c236-e28d-4331-9336-63a71d706ef0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:24.492518 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.492492 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzz5\" (UniqueName: \"kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5\") pod \"network-check-target-scdcl\" (UID: \"a132686f-9475-437d-826e-22fb93e86323\") " pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:24.492651 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:24.492613 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:46:24.492651 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:24.492631 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:46:24.492651 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:24.492642 2562 projected.go:196] Error preparing data for projected volume kube-api-access-cmzz5 for pod openshift-network-diagnostics/network-check-target-scdcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:24.492796 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:24.492697 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5 podName:a132686f-9475-437d-826e-22fb93e86323 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:25.492681705 +0000 UTC m=+4.174884876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmzz5" (UniqueName: "kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5") pod "network-check-target-scdcl" (UID: "a132686f-9475-437d-826e-22fb93e86323") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:24.743490 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.742936 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:41:23 +0000 UTC" deadline="2028-04-03 07:54:35.208502103 +0000 UTC" Oct 30 16:46:24.743490 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.743386 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12495h8m10.465124223s" Oct 30 16:46:24.797863 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.797830 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:24.798045 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:24.797967 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:24.798377 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.798355 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:24.798485 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:24.798463 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:24.810398 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.810310 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hfsmw" event={"ID":"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f","Type":"ContainerStarted","Data":"a7b568d421f6a612f9dd683fb4c69f0afba11581168228b603e39f73a03a9c6a"} Oct 30 16:46:24.815160 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.814698 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" event={"ID":"8216a0ebd9e778d471b8e472a27d4665","Type":"ContainerStarted","Data":"3425f923fe6b8c35b092bec775da87167619792193331d8d26f6d96c69a3ef78"} Oct 30 16:46:24.824950 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.824250 2562 generic.go:356] "Generic (PLEG): container finished" podID="b9b4913d9c6bf71319283ff1eade5a9e" containerID="d6323d2c4eb8c66f04c004580ed9ce2b3a678b1400e289509e24873faf812caf" exitCode=0 Oct 30 16:46:24.824950 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.824335 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" event={"ID":"b9b4913d9c6bf71319283ff1eade5a9e","Type":"ContainerDied","Data":"d6323d2c4eb8c66f04c004580ed9ce2b3a678b1400e289509e24873faf812caf"} Oct 30 16:46:24.835952 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.835923 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q9f8r" event={"ID":"b54c34b7-e8c3-4fe9-9b66-36336dfae405","Type":"ContainerStarted","Data":"1be7b177c8467d07515635775555327fa59b33db1f3cf2100c5f2e6ef349493d"} Oct 30 16:46:24.848353 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.848291 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l4rsf" event={"ID":"b174c80c-af6a-4f17-8442-1a029b286011","Type":"ContainerStarted","Data":"06bbf6f61b78f3ae3784d719972af3817a864582ffed7a1bd6f7e5b92d011189"} Oct 30 16:46:24.855175 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.855141 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nbbqf" event={"ID":"f8aeb454-5336-4fc1-8194-54d54a6cdb6c","Type":"ContainerStarted","Data":"2baff9f0fa3824f1aab8765d996070f22b46a05fe7497b4026d9d8798008ac70"} Oct 30 16:46:24.859728 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.859698 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" event={"ID":"252c7435-f4c1-4dac-a468-e3287c6a7e9e","Type":"ContainerStarted","Data":"0cb0b06adaba74a0251d98b36fe0ad9508165fa9cc4a372a21cfcef74e6f9967"} Oct 30 16:46:24.865188 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.865163 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4hlmr" event={"ID":"9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a","Type":"ContainerStarted","Data":"612ff75fe18ddb930ae5640e590053e6d754dd2a49f4c9f0a7259c66b8f97ff2"} Oct 30 16:46:24.872175 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.872114 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" event={"ID":"8c30c30a-8c1f-4552-b2d1-2692bc423d45","Type":"ContainerStarted","Data":"69975af90d98b8a7f922c1e5a7d172dd00cd1588810fe804987d9a300241d3aa"} Oct 30 16:46:24.882719 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.882629 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hh98" event={"ID":"a41c35af-f159-4802-9b53-fbb8cbee2104","Type":"ContainerStarted","Data":"d210d1881661386240975f06970c0bb1710c5358b3d15c73a1eb91b9f52dcbcf"} Oct 30 16:46:24.892331 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.892305 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" event={"ID":"852f74bb-c91e-4075-8335-aeba3e723709","Type":"ContainerStarted","Data":"035c8825d19d48aecd545e8171275583b3451316fc2fadf02d18814fa6f0a578"} Oct 30 16:46:24.907511 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:24.907283 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" podStartSLOduration=1.9072637989999999 podStartE2EDuration="1.907263799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:24.857995999 +0000 UTC m=+3.540199181" watchObservedRunningTime="2026-10-30 16:46:24.907263799 +0000 UTC m=+3.589466979" Oct 30 16:46:25.399498 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:25.399395 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:25.399498 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:25.399439 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:25.399668 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:25.399584 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:25.399668 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:25.399646 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs podName:3ca4c236-e28d-4331-9336-63a71d706ef0 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:27.399626137 +0000 UTC m=+6.081829309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs") pod "network-metrics-daemon-8mpgm" (UID: "3ca4c236-e28d-4331-9336-63a71d706ef0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:25.400074 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:25.400054 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:25.400142 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:25.400115 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret podName:ac8ed89d-9808-4bff-9d24-3e5fc34041ad nodeName:}" failed. No retries permitted until 2026-10-30 16:46:27.400097562 +0000 UTC m=+6.082300743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret") pod "global-pull-secret-syncer-wgxv6" (UID: "ac8ed89d-9808-4bff-9d24-3e5fc34041ad") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:25.501861 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:25.500477 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzz5\" (UniqueName: \"kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5\") pod \"network-check-target-scdcl\" (UID: \"a132686f-9475-437d-826e-22fb93e86323\") " pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:25.501861 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:25.500773 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:46:25.501861 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:25.500794 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:46:25.501861 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:25.500807 2562 projected.go:196] Error preparing data for projected volume kube-api-access-cmzz5 for pod openshift-network-diagnostics/network-check-target-scdcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:25.501861 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:25.500861 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5 podName:a132686f-9475-437d-826e-22fb93e86323 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:27.500844106 +0000 UTC m=+6.183047279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmzz5" (UniqueName: "kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5") pod "network-check-target-scdcl" (UID: "a132686f-9475-437d-826e-22fb93e86323") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:25.743804 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:25.743714 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:41:23 +0000 UTC" deadline="2028-08-14 18:56:35.899908148 +0000 UTC" Oct 30 16:46:25.743804 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:25.743758 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15698h10m10.156154492s" Oct 30 16:46:25.804770 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:25.801180 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:25.804770 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:25.801300 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:25.920042 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:25.919965 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" event={"ID":"b9b4913d9c6bf71319283ff1eade5a9e","Type":"ContainerStarted","Data":"af1e79e51e006f8e49c76b10060e39d9fdcb81b1f28a0648cc795e0c97928c03"} Oct 30 16:46:26.797673 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:26.797066 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:26.797673 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:26.797170 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:26.797673 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:26.797535 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:26.797673 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:26.797635 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:26.940931 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:26.940881 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hh98" event={"ID":"a41c35af-f159-4802-9b53-fbb8cbee2104","Type":"ContainerStarted","Data":"d18abfe4e19250e811e5f2e22d697b907716050fbc4a97e26d78c151866a6b34"} Oct 30 16:46:27.015884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:27.015824 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" podStartSLOduration=4.015806332 podStartE2EDuration="4.015806332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:25.964133089 +0000 UTC m=+4.646336272" watchObservedRunningTime="2026-10-30 16:46:27.015806332 +0000 UTC m=+5.698009513" Oct 30 16:46:27.420789 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:27.420752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:27.420789 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:27.420792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:27.421051 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:27.420922 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:27.421051 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:27.420987 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs podName:3ca4c236-e28d-4331-9336-63a71d706ef0 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:31.420967753 +0000 UTC m=+10.103170913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs") pod "network-metrics-daemon-8mpgm" (UID: "3ca4c236-e28d-4331-9336-63a71d706ef0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:27.421051 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:27.420922 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:27.421051 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:27.421034 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret podName:ac8ed89d-9808-4bff-9d24-3e5fc34041ad nodeName:}" failed. No retries permitted until 2026-10-30 16:46:31.421025971 +0000 UTC m=+10.103229129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret") pod "global-pull-secret-syncer-wgxv6" (UID: "ac8ed89d-9808-4bff-9d24-3e5fc34041ad") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:27.521660 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:27.521629 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzz5\" (UniqueName: \"kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5\") pod \"network-check-target-scdcl\" (UID: \"a132686f-9475-437d-826e-22fb93e86323\") " pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:27.521843 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:27.521765 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:46:27.521843 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:27.521780 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:46:27.521843 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:27.521788 2562 projected.go:196] Error preparing data for projected volume kube-api-access-cmzz5 for pod openshift-network-diagnostics/network-check-target-scdcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:27.521843 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:27.521831 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5 podName:a132686f-9475-437d-826e-22fb93e86323 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:31.521818858 +0000 UTC m=+10.204022029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmzz5" (UniqueName: "kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5") pod "network-check-target-scdcl" (UID: "a132686f-9475-437d-826e-22fb93e86323") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:27.799693 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:27.799664 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:27.800197 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:27.799773 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:27.954694 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:27.954657 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4hlmr" event={"ID":"9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a","Type":"ContainerStarted","Data":"21afb5c8014dac3a153209b2a6d00ea4c02ffdc7cdf226b44ac4b49eb4026731"} Oct 30 16:46:27.958705 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:27.958072 2562 generic.go:356] "Generic (PLEG): container finished" podID="a41c35af-f159-4802-9b53-fbb8cbee2104" containerID="d18abfe4e19250e811e5f2e22d697b907716050fbc4a97e26d78c151866a6b34" exitCode=0 Oct 30 16:46:27.958705 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:27.958123 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hh98" event={"ID":"a41c35af-f159-4802-9b53-fbb8cbee2104","Type":"ContainerDied","Data":"d18abfe4e19250e811e5f2e22d697b907716050fbc4a97e26d78c151866a6b34"} Oct 30 16:46:28.082479 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:28.082373 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4hlmr" podStartSLOduration=3.376945828 podStartE2EDuration="7.082354697s" totalImagesPullingTime="3.705408869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:28.014835846 +0000 UTC m=+6.697039025" watchObservedRunningTime="2026-10-30 16:46:28.082354697 +0000 UTC m=+6.764557878" Oct 30 16:46:28.797881 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:28.797812 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:28.798039 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:28.797956 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:28.798378 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:28.798358 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:28.798519 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:28.798466 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:28.964690 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:28.964656 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hfsmw" event={"ID":"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f","Type":"ContainerStarted","Data":"1dd1840ef33eed40fc5803db60f81f68a9706f4bf14d17689e845e14ce392eac"} Oct 30 16:46:29.008129 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:29.008071 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hfsmw" podStartSLOduration=3.545386874 podStartE2EDuration="8.008051686s" totalImagesPullingTime="4.462664812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:29.007083156 +0000 UTC m=+7.689286340" watchObservedRunningTime="2026-10-30 16:46:29.008051686 +0000 UTC m=+7.690254866" Oct 30 16:46:29.230619 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:29.230586 2562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:46:29.231071 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:29.231049 2562 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:46:29.799530 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:29.799501 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:29.799690 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:29.799626 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:29.969627 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:29.969561 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" event={"ID":"252c7435-f4c1-4dac-a468-e3287c6a7e9e","Type":"ContainerStarted","Data":"70e2b107caefbc00f814da9487773818f1c5225603e82a0847d488e9b6dfdec7"} Oct 30 16:46:30.797976 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:30.797508 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:30.797976 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:30.797643 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:30.798179 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:30.798044 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:30.798179 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:30.798120 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:30.971342 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:30.971314 2562 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:46:31.114393 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:31.114367 2562 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:46:31.454105 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:31.454070 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:31.454288 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:31.454120 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:31.454288 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:31.454254 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:31.454384 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:31.454314 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs podName:3ca4c236-e28d-4331-9336-63a71d706ef0 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:39.45429544 +0000 UTC m=+18.136498611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs") pod "network-metrics-daemon-8mpgm" (UID: "3ca4c236-e28d-4331-9336-63a71d706ef0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:31.454444 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:31.454385 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:31.454444 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:31.454418 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret podName:ac8ed89d-9808-4bff-9d24-3e5fc34041ad nodeName:}" failed. No retries permitted until 2026-10-30 16:46:39.454407552 +0000 UTC m=+18.136610715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret") pod "global-pull-secret-syncer-wgxv6" (UID: "ac8ed89d-9808-4bff-9d24-3e5fc34041ad") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:31.554525 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:31.554489 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzz5\" (UniqueName: \"kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5\") pod \"network-check-target-scdcl\" (UID: \"a132686f-9475-437d-826e-22fb93e86323\") " pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:31.554705 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:31.554687 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:46:31.559014 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:31.554710 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:46:31.559014 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:31.554722 2562 projected.go:196] Error preparing data for projected volume kube-api-access-cmzz5 for pod openshift-network-diagnostics/network-check-target-scdcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:31.559014 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:31.554776 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5 podName:a132686f-9475-437d-826e-22fb93e86323 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:39.554762191 +0000 UTC m=+18.236965350 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmzz5" (UniqueName: "kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5") pod "network-check-target-scdcl" (UID: "a132686f-9475-437d-826e-22fb93e86323") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:31.742091 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:31.741980 2562 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:46:31.11439036Z","UUID":"0cab6183-8850-4423-8336-1e656698761d","Handler":null,"Name":"","Endpoint":""} Oct 30 16:46:31.743804 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:31.743766 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:46:31.743804 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:31.743799 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:46:31.799943 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:31.799914 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:31.800081 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:31.800024 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:31.976059 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:31.976024 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q9f8r" event={"ID":"b54c34b7-e8c3-4fe9-9b66-36336dfae405","Type":"ContainerStarted","Data":"ee4f204dbe3601ef9f924cb129bfb51488a4f73a8c93381da58f767d5bbd23b5"} Oct 30 16:46:31.979034 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:31.978890 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nbbqf" event={"ID":"f8aeb454-5336-4fc1-8194-54d54a6cdb6c","Type":"ContainerStarted","Data":"028e38e75060e9d2e75778b11fc554f608e6330984adf32025a4c06831f8398e"} Oct 30 16:46:31.981113 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:31.981090 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" event={"ID":"252c7435-f4c1-4dac-a468-e3287c6a7e9e","Type":"ContainerStarted","Data":"58347f368f133e4620175ee9e31c9569647190dac2d3c671469df00d5cd7bd86"} Oct 30 16:46:32.073455 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:32.073355 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-q9f8r" podStartSLOduration=2.81326622 podStartE2EDuration="10.073337014s" totalImagesPullingTime="7.260070794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:32.031336755 +0000 UTC m=+10.713539937" watchObservedRunningTime="2026-10-30 16:46:32.073337014 +0000 UTC m=+10.755540195" Oct 30 16:46:32.797854 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:32.797603 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:32.798056 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:32.797980 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:32.798056 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:32.797988 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:32.798142 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:32.798085 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:32.985720 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:32.985636 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" event={"ID":"252c7435-f4c1-4dac-a468-e3287c6a7e9e","Type":"ContainerStarted","Data":"dfb1dad26a46926a1505bd62138cddf63c0b0a431ea7edf26b095805ef80258e"} Oct 30 16:46:32.988159 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:32.988126 2562 generic.go:356] "Generic (PLEG): container finished" podID="a41c35af-f159-4802-9b53-fbb8cbee2104" containerID="6d32e95e901f83fd7efd2343a32e18c5cea99a687d713c566510ef38715b5ab8" exitCode=0 Oct 30 16:46:32.988293 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:32.988206 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hh98" event={"ID":"a41c35af-f159-4802-9b53-fbb8cbee2104","Type":"ContainerDied","Data":"6d32e95e901f83fd7efd2343a32e18c5cea99a687d713c566510ef38715b5ab8"} Oct 30 16:46:33.031888 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:33.031813 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" podStartSLOduration=4.151682411 podStartE2EDuration="12.031799067s" totalImagesPullingTime="7.880116656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:33.031284181 +0000 UTC m=+11.713487362" watchObservedRunningTime="2026-10-30 16:46:33.031799067 +0000 UTC m=+11.714002247" Oct 30 16:46:33.032161 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:33.032139 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-nbbqf" podStartSLOduration=3.361070411 podStartE2EDuration="11.032131569s" totalImagesPullingTime="7.671061158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:32.073436739 +0000 UTC m=+10.755639947" watchObservedRunningTime="2026-10-30 16:46:33.032131569 +0000 UTC m=+11.714334749" Oct 30 16:46:33.797980 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:33.797954 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:33.798116 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:33.798070 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:33.992023 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:33.991938 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l4rsf" event={"ID":"b174c80c-af6a-4f17-8442-1a029b286011","Type":"ContainerStarted","Data":"03ee8f6db9d448fad471b1e959102225c5da540ecdf18425005a3f49f2cf3d63"} Oct 30 16:46:33.993801 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:33.993775 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" event={"ID":"8c30c30a-8c1f-4552-b2d1-2692bc423d45","Type":"ContainerStarted","Data":"41a1807a0fdb517bb8759f3fed86d92f5ded3a151d4b17f6ab0c98c778e6e341"} Oct 30 16:46:33.996199 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:33.996177 2562 generic.go:356] "Generic (PLEG): container finished" podID="a41c35af-f159-4802-9b53-fbb8cbee2104" containerID="38d3745e75913fcefb6e44b822e1ff32e70e6c1e75c2ff0280de9c183ec57a8b" exitCode=0 Oct 30 16:46:33.996321 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:33.996249 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hh98" event={"ID":"a41c35af-f159-4802-9b53-fbb8cbee2104","Type":"ContainerDied","Data":"38d3745e75913fcefb6e44b822e1ff32e70e6c1e75c2ff0280de9c183ec57a8b"} Oct 30 16:46:34.035941 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:34.035868 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l4rsf" podStartSLOduration=4.460068907 podStartE2EDuration="12.035856188s" totalImagesPullingTime="7.575787281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:34.035521191 +0000 UTC m=+12.717724373" watchObservedRunningTime="2026-10-30 16:46:34.035856188 +0000 UTC m=+12.718059367" Oct 30 16:46:34.132059 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:34.132001 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" podStartSLOduration=3.4015169419999998 podStartE2EDuration="13.131983478s" totalImagesPullingTime="9.730466536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:34.131888778 +0000 UTC m=+12.814091957" watchObservedRunningTime="2026-10-30 16:46:34.131983478 +0000 UTC m=+12.814186660" Oct 30 16:46:34.797424 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:34.797395 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:34.797560 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:34.797442 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:34.797560 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:34.797522 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:34.797685 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:34.797593 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:35.001690 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:35.001605 2562 generic.go:356] "Generic (PLEG): container finished" podID="a41c35af-f159-4802-9b53-fbb8cbee2104" containerID="93c5cc87c39b5c64f18b6e6a18eb092243d33bb1e4c8c8206b63cac1a91ec277" exitCode=0 Oct 30 16:46:35.001690 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:35.001671 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hh98" event={"ID":"a41c35af-f159-4802-9b53-fbb8cbee2104","Type":"ContainerDied","Data":"93c5cc87c39b5c64f18b6e6a18eb092243d33bb1e4c8c8206b63cac1a91ec277"} Oct 30 16:46:35.797931 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:35.797879 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:35.798098 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:35.798026 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:36.797474 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:36.797441 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:36.797990 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:36.797548 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:36.797990 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:36.797597 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:36.797990 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:36.797703 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:37.009375 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:37.009136 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hh98" event={"ID":"a41c35af-f159-4802-9b53-fbb8cbee2104","Type":"ContainerStarted","Data":"7425163bdce5c2faa90c91f3aab24425b8adc382d0729316790af2edbcfea31f"} Oct 30 16:46:37.797782 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:37.797746 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:37.798309 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:37.797857 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:38.013852 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:38.013812 2562 generic.go:356] "Generic (PLEG): container finished" podID="a41c35af-f159-4802-9b53-fbb8cbee2104" containerID="7425163bdce5c2faa90c91f3aab24425b8adc382d0729316790af2edbcfea31f" exitCode=0 Oct 30 16:46:38.014017 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:38.013875 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hh98" event={"ID":"a41c35af-f159-4802-9b53-fbb8cbee2104","Type":"ContainerDied","Data":"7425163bdce5c2faa90c91f3aab24425b8adc382d0729316790af2edbcfea31f"} Oct 30 16:46:38.797983 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:38.797950 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:38.798428 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:38.797992 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:38.798428 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:38.798052 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:38.798428 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:38.798196 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:39.018728 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.018649 2562 generic.go:356] "Generic (PLEG): container finished" podID="a41c35af-f159-4802-9b53-fbb8cbee2104" containerID="1e1ad57e69290cb7a4aaf9f2cc0733e63f72524c478c3f7622be0e0d2ea15ac1" exitCode=0 Oct 30 16:46:39.018728 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.018711 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hh98" event={"ID":"a41c35af-f159-4802-9b53-fbb8cbee2104","Type":"ContainerDied","Data":"1e1ad57e69290cb7a4aaf9f2cc0733e63f72524c478c3f7622be0e0d2ea15ac1"} Oct 30 16:46:39.021497 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.021478 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" event={"ID":"852f74bb-c91e-4075-8335-aeba3e723709","Type":"ContainerStarted","Data":"7b95915ee7eb5d30d1d70e3b27e15710e2af6971d6d307513f79ee4e33f37a5f"} Oct 30 16:46:39.021555 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.021531 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" event={"ID":"852f74bb-c91e-4075-8335-aeba3e723709","Type":"ContainerStarted","Data":"c22145890cad704c7ca8bdd5f6022ca5655f9c028d92222524bd5444c9dfb7c8"} Oct 30 16:46:39.021594 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.021562 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" event={"ID":"852f74bb-c91e-4075-8335-aeba3e723709","Type":"ContainerStarted","Data":"f1f3a9215f9f3a3040a741f51241a23c80706edf7111c0c6621412fd707e5d44"} Oct 30 16:46:39.021594 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.021589 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" event={"ID":"852f74bb-c91e-4075-8335-aeba3e723709","Type":"ContainerStarted","Data":"1998ce43ca361d29ce893fe5568d5e4a2ba4ec35b2cddffedd8ab137e34fe514"} Oct 30 16:46:39.021682 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.021616 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" event={"ID":"852f74bb-c91e-4075-8335-aeba3e723709","Type":"ContainerStarted","Data":"0f8cc84e3eb79b0ee114f80a54c35c09e8a4578fad5ece1b3c6a8847257d740a"} Oct 30 16:46:39.021682 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.021642 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" event={"ID":"852f74bb-c91e-4075-8335-aeba3e723709","Type":"ContainerStarted","Data":"8093f825099929b71cd90967e8f1ecf39397becefec5899b3087ce5b34e62603"} Oct 30 16:46:39.330401 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.330318 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:46:39.330553 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.330466 2562 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:46:39.330971 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.330947 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:46:39.515637 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.515602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:39.515637 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.515641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:39.515869 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:39.515736 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:39.515869 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:39.515749 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:39.515869 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:39.515798 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret podName:ac8ed89d-9808-4bff-9d24-3e5fc34041ad nodeName:}" failed. No retries permitted until 2026-10-30 16:46:55.515782392 +0000 UTC m=+34.197985550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret") pod "global-pull-secret-syncer-wgxv6" (UID: "ac8ed89d-9808-4bff-9d24-3e5fc34041ad") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:46:39.515869 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:39.515812 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs podName:3ca4c236-e28d-4331-9336-63a71d706ef0 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:55.515805602 +0000 UTC m=+34.198008761 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs") pod "network-metrics-daemon-8mpgm" (UID: "3ca4c236-e28d-4331-9336-63a71d706ef0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:46:39.616085 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.615995 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzz5\" (UniqueName: \"kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5\") pod \"network-check-target-scdcl\" (UID: \"a132686f-9475-437d-826e-22fb93e86323\") " pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:39.616233 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:39.616184 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:46:39.616233 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:39.616207 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:46:39.616233 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:39.616219 2562 projected.go:196] Error preparing data for projected volume kube-api-access-cmzz5 for pod openshift-network-diagnostics/network-check-target-scdcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:39.616382 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:39.616284 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5 podName:a132686f-9475-437d-826e-22fb93e86323 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:55.61626541 +0000 UTC m=+34.298468583 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmzz5" (UniqueName: "kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5") pod "network-check-target-scdcl" (UID: "a132686f-9475-437d-826e-22fb93e86323") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:46:39.798003 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:39.797965 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:39.798691 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:39.798064 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:40.027020 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:40.026940 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2hh98" event={"ID":"a41c35af-f159-4802-9b53-fbb8cbee2104","Type":"ContainerStarted","Data":"4be372c12c728193f8bfac198a1a033e1c98e4ef7a63491932657b7828900a69"} Oct 30 16:46:40.076994 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:40.076939 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2hh98" podStartSLOduration=8.147025627 podStartE2EDuration="18.076925212s" totalImagesPullingTime="9.929899585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:40.07625433 +0000 UTC m=+18.758457509" watchObservedRunningTime="2026-10-30 16:46:40.076925212 +0000 UTC m=+18.759128392" Oct 30 16:46:40.797830 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:40.797796 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:40.797975 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:40.797796 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:40.797975 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:40.797940 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:40.798056 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:40.797991 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:41.034159 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:41.034080 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" event={"ID":"852f74bb-c91e-4075-8335-aeba3e723709","Type":"ContainerStarted","Data":"e5132e96dcc29bdd0aaffd51a807d053dd032c80a7da27e23f124f0a8ddf359c"} Oct 30 16:46:41.798728 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:41.798688 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:41.799196 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:41.798793 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:42.797461 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:42.797436 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:42.797569 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:42.797540 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:42.797569 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:42.797565 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:42.797644 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:42.797624 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:43.041820 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:43.041557 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" event={"ID":"852f74bb-c91e-4075-8335-aeba3e723709","Type":"ContainerStarted","Data":"c6213f1b618d8259e61d888e6682282f30410905ef6122398d406cb6b73bda65"} Oct 30 16:46:43.041820 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:43.041798 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:43.042529 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:43.041844 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:43.057081 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:43.057052 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:43.135995 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:43.135936 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" podStartSLOduration=6.893248011 podStartE2EDuration="21.135895667s" totalImagesPullingTime="14.242647656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:43.135494083 +0000 UTC m=+21.817697262" watchObservedRunningTime="2026-10-30 16:46:43.135895667 +0000 UTC m=+21.818098848" Oct 30 16:46:43.797424 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:43.797396 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:43.797590 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:43.797501 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:44.043957 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:44.043924 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:44.058666 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:44.058594 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:46:44.798042 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:44.798011 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:44.798220 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:44.798122 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:44.798220 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:44.798179 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:44.798322 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:44.798281 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:44.926810 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:44.926783 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wgxv6"] Oct 30 16:46:44.929393 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:44.929366 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-scdcl"] Oct 30 16:46:44.929529 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:44.929501 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:44.929633 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:44.929609 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:44.930096 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:44.930073 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8mpgm"] Oct 30 16:46:45.045284 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:45.045256 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:45.045646 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:45.045258 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:45.045646 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:45.045382 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:45.045646 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:45.045433 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:46.797300 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:46.797264 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:46.797300 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:46.797302 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:46.797885 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:46.797381 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:46.797885 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:46.797409 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:46.797885 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:46.797472 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:46.797885 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:46.797548 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:48.798083 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:48.798049 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:48.798083 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:48.798080 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:48.798675 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:48.798230 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:48.798675 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:48.798268 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:48.798675 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:48.798352 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:48.798675 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:48.798415 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:50.797469 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:50.797434 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:50.797945 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:50.797550 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:50.797945 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:50.797580 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:50.797945 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:50.797568 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wgxv6" podUID="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" Oct 30 16:46:50.797945 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:50.797683 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mpgm" podUID="3ca4c236-e28d-4331-9336-63a71d706ef0" Oct 30 16:46:50.797945 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:50.797748 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scdcl" podUID="a132686f-9475-437d-826e-22fb93e86323" Oct 30 16:46:51.153203 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.153023 2562 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeReady" Oct 30 16:46:51.153348 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.153309 2562 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 16:46:51.207511 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.207468 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fb8qf"] Oct 30 16:46:51.215360 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.215337 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mmhr5"] Oct 30 16:46:51.215477 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.215458 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fb8qf" Oct 30 16:46:51.218081 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.218060 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-797d7cb6cf-rzmrh"] Oct 30 16:46:51.218187 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.218171 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mmhr5" Oct 30 16:46:51.220731 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.220706 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mhsj5"] Oct 30 16:46:51.221006 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.220984 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.224021 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.224002 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56"] Oct 30 16:46:51.224137 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.224121 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mhsj5" Oct 30 16:46:51.229031 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.229009 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5c78887cdf-9jhzv"] Oct 30 16:46:51.229140 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.229123 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56" Oct 30 16:46:51.235879 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.235858 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45"] Oct 30 16:46:51.236022 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.236006 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.240199 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.240182 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2"] Oct 30 16:46:51.240353 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.240323 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:46:51.245117 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.245098 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jmx89"] Oct 30 16:46:51.245218 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.245202 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:46:51.247670 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.247653 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 16:46:51.247969 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.247953 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 16:46:51.252219 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.252198 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq"] Oct 30 16:46:51.252428 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.252404 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.259061 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.258990 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52"] Oct 30 16:46:51.259201 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.259103 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:46:51.262178 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.262156 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 16:46:51.262578 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.262554 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 16:46:51.262984 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.262961 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-76ckj\"" Oct 30 16:46:51.263073 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.263047 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 16:46:51.263209 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.263157 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zs99n\"" Oct 30 16:46:51.263333 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.263259 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 16:46:51.266843 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.266823 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6rrfj"] Oct 30 16:46:51.269927 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.267932 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:46:51.272675 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.272652 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67785c878-8c9pr"] Oct 30 16:46:51.273034 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.273011 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:51.276964 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.276943 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mhsj5"] Oct 30 16:46:51.277061 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.276985 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5"] Oct 30 16:46:51.277111 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.277088 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.281944 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.281921 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tb2dr"] Oct 30 16:46:51.282330 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.282312 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5" Oct 30 16:46:51.286518 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.286497 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fb8qf"] Oct 30 16:46:51.286612 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.286568 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-n758f"] Oct 30 16:46:51.286665 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.286623 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tb2dr" Oct 30 16:46:51.294018 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.293990 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7zrs4"] Oct 30 16:46:51.295221 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.295195 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.297972 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.297950 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mmhr5"] Oct 30 16:46:51.298077 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.297990 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56"] Oct 30 16:46:51.298077 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298013 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-797d7cb6cf-rzmrh"] Oct 30 16:46:51.298077 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298031 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jmx89"] Oct 30 16:46:51.298077 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298049 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c78887cdf-9jhzv"] Oct 30 16:46:51.298077 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298068 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52"] Oct 30 16:46:51.298322 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298095 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45"] Oct 30 16:46:51.298322 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298115 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq"] Oct 30 16:46:51.298322 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298133 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-n758f"] Oct 30 16:46:51.298322 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298212 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:46:51.298692 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298670 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4x9q\" (UniqueName: \"kubernetes.io/projected/159cde9b-22a5-4930-9bf6-2535ab4af3d4-kube-api-access-p4x9q\") pod \"kas-connection-checker-685895c7c9-mhsj5\" (UID: \"159cde9b-22a5-4930-9bf6-2535ab4af3d4\") " pod="kube-system/kas-connection-checker-685895c7c9-mhsj5" Oct 30 16:46:51.298762 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298716 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-certificates\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.298762 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298746 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-default-certificate\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.298881 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298770 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-installation-pull-secrets\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.298881 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298800 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2"] Oct 30 16:46:51.298881 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298807 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tnhb\" (UniqueName: \"kubernetes.io/projected/30001919-517f-4031-9ef4-bc0fbb43d122-kube-api-access-4tnhb\") pod \"kas-connection-checker-685895c7c9-mmhr5\" (UID: \"30001919-517f-4031-9ef4-bc0fbb43d122\") " pod="kube-system/kas-connection-checker-685895c7c9-mmhr5" Oct 30 16:46:51.298881 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298830 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.299120 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.298994 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.299120 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.299091 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.299217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.299121 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-trusted-ca\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.299217 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.299177 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-stats-auth\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.299480 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.299447 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgpkr\" (UniqueName: \"kubernetes.io/projected/3095c033-c4ff-42cd-9530-1be17a144a31-kube-api-access-fgpkr\") pod \"kas-connection-checker-685895c7c9-fb8qf\" (UID: \"3095c033-c4ff-42cd-9530-1be17a144a31\") " pod="kube-system/kas-connection-checker-685895c7c9-fb8qf" Oct 30 16:46:51.299591 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.299511 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d38312b4-bd6e-4a5f-8a33-d93fb827debc-ca-trust-extracted\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.299591 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.299528 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-bound-sa-token\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.299591 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.299543 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q52w4\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-kube-api-access-q52w4\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.299591 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.299577 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjrlc\" (UniqueName: \"kubernetes.io/projected/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-kube-api-access-pjrlc\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.299804 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.299594 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-image-registry-private-configuration\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.299804 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.299618 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4srgz\" (UniqueName: \"kubernetes.io/projected/d7a04182-8031-4fa8-8bed-4f8b60b6cc97-kube-api-access-4srgz\") pod \"volume-data-source-validator-58c9f5f46d-ggh56\" (UID: \"d7a04182-8031-4fa8-8bed-4f8b60b6cc97\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56" Oct 30 16:46:51.300733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.300676 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67785c878-8c9pr"] Oct 30 16:46:51.301689 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.301666 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tb2dr"] Oct 30 16:46:51.302596 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.302568 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6rrfj"] Oct 30 16:46:51.314661 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.314635 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7zrs4"] Oct 30 16:46:51.322335 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.322306 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5"] Oct 30 16:46:51.348734 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.348710 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 16:46:51.349057 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.348912 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 16:46:51.349057 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349014 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:46:51.349178 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349159 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:46:51.349234 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349208 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 16:46:51.349328 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349313 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:46:51.349422 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349409 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:46:51.349485 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349434 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 16:46:51.349485 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349478 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:46:51.349596 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349519 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 16:46:51.349596 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349533 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 16:46:51.349596 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349571 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 16:46:51.349720 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349644 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:46:51.349720 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349707 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 16:46:51.349801 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349730 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 16:46:51.349845 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349829 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:46:51.349894 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349848 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 16:46:51.349894 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349858 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 16:46:51.349894 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349855 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:46:51.350024 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349952 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 16:46:51.350024 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.349982 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 16:46:51.350024 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.350002 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 16:46:51.350024 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.350008 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:46:51.350143 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.350029 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:46:51.350143 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.350067 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:46:51.350514 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.350499 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:46:51.350665 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.350653 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:46:51.350869 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.350850 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:46:51.362506 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.362485 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 16:46:51.362895 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.362879 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 16:46:51.362998 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.362980 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 16:46:51.363055 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363006 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 16:46:51.363112 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363055 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 16:46:51.363112 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363071 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 16:46:51.363112 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363088 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 16:46:51.363112 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363105 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 16:46:51.363252 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363117 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mcfkl\"" Oct 30 16:46:51.363252 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363133 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 16:46:51.363475 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363460 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 16:46:51.363546 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363477 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 16:46:51.363546 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363515 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:46:51.363546 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363538 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 16:46:51.363684 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363605 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pjmpr\"" Oct 30 16:46:51.363684 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363667 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2qtkg\"" Oct 30 16:46:51.363781 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363716 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-g8gt8\"" Oct 30 16:46:51.363781 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363723 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:46:51.363781 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363745 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-b86vs\"" Oct 30 16:46:51.363781 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363750 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8hr2j\"" Oct 30 16:46:51.363781 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363764 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nwhz8\"" Oct 30 16:46:51.363992 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363792 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9wpv\"" Oct 30 16:46:51.363992 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363849 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2twsj\"" Oct 30 16:46:51.363992 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363879 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ffzh4\"" Oct 30 16:46:51.363992 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363933 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 16:46:51.363992 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.363959 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-c2bll\"" Oct 30 16:46:51.400115 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400085 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-installation-pull-secrets\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.400280 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400123 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-bound-sa-token\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.400280 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400147 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.400280 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b516f174-836e-4711-8999-80ce5a592e28-serving-cert\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.400280 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-image-registry-private-configuration\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.400487 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400381 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b516f174-836e-4711-8999-80ce5a592e28-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.400487 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400447 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dacad254-b0c1-48fd-8377-1067d65ea230-serving-cert\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.400487 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400478 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srgz\" (UniqueName: \"kubernetes.io/projected/d7a04182-8031-4fa8-8bed-4f8b60b6cc97-kube-api-access-4srgz\") pod \"volume-data-source-validator-58c9f5f46d-ggh56\" (UID: \"d7a04182-8031-4fa8-8bed-4f8b60b6cc97\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56" Oct 30 16:46:51.400627 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400502 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:46:51.400627 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400550 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-default-certificate\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.400704 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400651 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-image-registry-private-configuration\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.400751 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400716 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmgvx\" (UniqueName: \"kubernetes.io/projected/b34094c9-7349-4284-8b86-2253f9e4d87f-kube-api-access-zmgvx\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:46:51.400821 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35a14012-73eb-4f3b-b762-f56c6a963e44-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:46:51.400874 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400836 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b516f174-836e-4711-8999-80ce5a592e28-tmp\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.400874 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400862 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:46:51.400986 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400919 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:51.401024 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.400993 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm5w2\" (UniqueName: \"kubernetes.io/projected/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-kube-api-access-gm5w2\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:46:51.401084 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401062 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnhb\" (UniqueName: \"kubernetes.io/projected/30001919-517f-4031-9ef4-bc0fbb43d122-kube-api-access-4tnhb\") pod \"kas-connection-checker-685895c7c9-mmhr5\" (UID: \"30001919-517f-4031-9ef4-bc0fbb43d122\") " pod="kube-system/kas-connection-checker-685895c7c9-mmhr5" Oct 30 16:46:51.401128 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401098 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b516f174-836e-4711-8999-80ce5a592e28-service-ca-bundle\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.401184 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.401184 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.401184 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401172 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqff6\" (UniqueName: \"kubernetes.io/projected/f13fd623-29b9-4d8b-b76d-668ca123010f-kube-api-access-sqff6\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:51.401318 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401206 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8f6w\" (UniqueName: \"kubernetes.io/projected/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-kube-api-access-z8f6w\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:46:51.401318 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.401274 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:46:51.401318 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.401288 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d7cb6cf-rzmrh: secret "image-registry-tls" not found Oct 30 16:46:51.401318 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.401274 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:46:51.401494 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401336 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b516f174-836e-4711-8999-80ce5a592e28-snapshots\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.401494 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.401347 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls podName:d38312b4-bd6e-4a5f-8a33-d93fb827debc nodeName:}" failed. No retries permitted until 2026-10-30 16:46:51.901327048 +0000 UTC m=+30.583530221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls") pod "image-registry-797d7cb6cf-rzmrh" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc") : secret "image-registry-tls" not found Oct 30 16:46:51.401494 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.401383 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:51.901365966 +0000 UTC m=+30.583569133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : secret "router-metrics-certs-default" not found Oct 30 16:46:51.401494 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401409 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q52w4\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-kube-api-access-q52w4\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.401494 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401451 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgpkr\" (UniqueName: \"kubernetes.io/projected/3095c033-c4ff-42cd-9530-1be17a144a31-kube-api-access-fgpkr\") pod \"kas-connection-checker-685895c7c9-fb8qf\" (UID: \"3095c033-c4ff-42cd-9530-1be17a144a31\") " pod="kube-system/kas-connection-checker-685895c7c9-fb8qf" Oct 30 16:46:51.401494 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401482 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d38312b4-bd6e-4a5f-8a33-d93fb827debc-ca-trust-extracted\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.401777 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-bound-sa-token\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.401777 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35a14012-73eb-4f3b-b762-f56c6a963e44-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:46:51.401777 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401659 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrlc\" (UniqueName: \"kubernetes.io/projected/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-kube-api-access-pjrlc\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.401777 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401687 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:46:51.401777 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401759 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6zs9\" (UniqueName: \"kubernetes.io/projected/b516f174-836e-4711-8999-80ce5a592e28-kube-api-access-q6zs9\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.402062 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401796 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68321123-f8fc-4c6a-bd22-1f614a8a3b55-cert\") pod \"ingress-canary-tb2dr\" (UID: \"68321123-f8fc-4c6a-bd22-1f614a8a3b55\") " pod="openshift-ingress-canary/ingress-canary-tb2dr" Oct 30 16:46:51.402062 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401845 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qprsq\" (UniqueName: \"kubernetes.io/projected/68321123-f8fc-4c6a-bd22-1f614a8a3b55-kube-api-access-qprsq\") pod \"ingress-canary-tb2dr\" (UID: \"68321123-f8fc-4c6a-bd22-1f614a8a3b55\") " pod="openshift-ingress-canary/ingress-canary-tb2dr" Oct 30 16:46:51.402062 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f13fd623-29b9-4d8b-b76d-668ca123010f-config-volume\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:51.402062 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401966 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d38312b4-bd6e-4a5f-8a33-d93fb827debc-ca-trust-extracted\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.402062 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.401994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4x9q\" (UniqueName: \"kubernetes.io/projected/159cde9b-22a5-4930-9bf6-2535ab4af3d4-kube-api-access-p4x9q\") pod \"kas-connection-checker-685895c7c9-mhsj5\" (UID: \"159cde9b-22a5-4930-9bf6-2535ab4af3d4\") " pod="kube-system/kas-connection-checker-685895c7c9-mhsj5" Oct 30 16:46:51.402062 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-certificates\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.402062 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402056 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:46:51.402324 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402081 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f13fd623-29b9-4d8b-b76d-668ca123010f-tmp-dir\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:51.402324 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402107 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xjxv\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-kube-api-access-7xjxv\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.402324 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402135 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-installation-pull-secrets\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.402324 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402167 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dacad254-b0c1-48fd-8377-1067d65ea230-trusted-ca\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.402324 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402223 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cmkh\" (UniqueName: \"kubernetes.io/projected/dacad254-b0c1-48fd-8377-1067d65ea230-kube-api-access-8cmkh\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.402324 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402276 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3794a50e-6c2f-40bc-9be7-eda1388f3178-ca-trust-extracted\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.402324 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402297 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-certificates\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.402324 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402312 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgkdb\" (UniqueName: \"kubernetes.io/projected/35a14012-73eb-4f3b-b762-f56c6a963e44-kube-api-access-sgkdb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:46:51.402669 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-trusted-ca\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.402669 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402347 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.402669 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402406 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-trusted-ca\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.402669 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402447 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:46:51.402669 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.402502 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:51.902487865 +0000 UTC m=+30.584691044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : configmap references non-existent config key: service-ca.crt Oct 30 16:46:51.402669 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402509 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-certificates\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.402669 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402518 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64l24\" (UniqueName: \"kubernetes.io/projected/3cdefc81-0135-4053-bc7b-78958686e8ea-kube-api-access-64l24\") pod \"network-check-source-6f78d59f99-jdsr5\" (UID: \"3cdefc81-0135-4053-bc7b-78958686e8ea\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5" Oct 30 16:46:51.402669 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-stats-auth\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.402669 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402592 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-config\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:46:51.402669 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402619 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dacad254-b0c1-48fd-8377-1067d65ea230-config\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.402669 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402643 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-serving-cert\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:46:51.403139 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.402750 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:46:51.403416 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.403378 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-trusted-ca\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.404989 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.404970 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-installation-pull-secrets\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.404989 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.404978 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-default-certificate\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.404989 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.404982 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-stats-auth\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.405127 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.405108 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-image-registry-private-configuration\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.434275 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.434241 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4x9q\" (UniqueName: \"kubernetes.io/projected/159cde9b-22a5-4930-9bf6-2535ab4af3d4-kube-api-access-p4x9q\") pod \"kas-connection-checker-685895c7c9-mhsj5\" (UID: \"159cde9b-22a5-4930-9bf6-2535ab4af3d4\") " pod="kube-system/kas-connection-checker-685895c7c9-mhsj5" Oct 30 16:46:51.435498 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.435464 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srgz\" (UniqueName: \"kubernetes.io/projected/d7a04182-8031-4fa8-8bed-4f8b60b6cc97-kube-api-access-4srgz\") pod \"volume-data-source-validator-58c9f5f46d-ggh56\" (UID: \"d7a04182-8031-4fa8-8bed-4f8b60b6cc97\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56" Oct 30 16:46:51.435611 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.435586 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgpkr\" (UniqueName: \"kubernetes.io/projected/3095c033-c4ff-42cd-9530-1be17a144a31-kube-api-access-fgpkr\") pod \"kas-connection-checker-685895c7c9-fb8qf\" (UID: \"3095c033-c4ff-42cd-9530-1be17a144a31\") " pod="kube-system/kas-connection-checker-685895c7c9-fb8qf" Oct 30 16:46:51.436314 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.436293 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-bound-sa-token\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.439692 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.439668 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q52w4\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-kube-api-access-q52w4\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.440249 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.440223 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnhb\" (UniqueName: \"kubernetes.io/projected/30001919-517f-4031-9ef4-bc0fbb43d122-kube-api-access-4tnhb\") pod \"kas-connection-checker-685895c7c9-mmhr5\" (UID: \"30001919-517f-4031-9ef4-bc0fbb43d122\") " pod="kube-system/kas-connection-checker-685895c7c9-mmhr5" Oct 30 16:46:51.441122 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.441107 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrlc\" (UniqueName: \"kubernetes.io/projected/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-kube-api-access-pjrlc\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.503174 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503134 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qprsq\" (UniqueName: \"kubernetes.io/projected/68321123-f8fc-4c6a-bd22-1f614a8a3b55-kube-api-access-qprsq\") pod \"ingress-canary-tb2dr\" (UID: \"68321123-f8fc-4c6a-bd22-1f614a8a3b55\") " pod="openshift-ingress-canary/ingress-canary-tb2dr" Oct 30 16:46:51.503174 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f13fd623-29b9-4d8b-b76d-668ca123010f-config-volume\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:51.503384 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503196 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:46:51.503384 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503266 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f13fd623-29b9-4d8b-b76d-668ca123010f-tmp-dir\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:51.503384 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xjxv\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-kube-api-access-7xjxv\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.503384 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.503273 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:46:51.503384 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dacad254-b0c1-48fd-8377-1067d65ea230-trusted-ca\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.503384 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503357 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cmkh\" (UniqueName: \"kubernetes.io/projected/dacad254-b0c1-48fd-8377-1067d65ea230-kube-api-access-8cmkh\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.503384 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.503385 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert podName:7918c4b3-b4f4-4435-9de9-77fbe4906a7f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:52.003362487 +0000 UTC m=+30.685565657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7zrs4" (UID: "7918c4b3-b4f4-4435-9de9-77fbe4906a7f") : secret "networking-console-plugin-cert" not found Oct 30 16:46:51.503733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503418 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3794a50e-6c2f-40bc-9be7-eda1388f3178-ca-trust-extracted\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.503733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503443 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-certificates\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.503733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgkdb\" (UniqueName: \"kubernetes.io/projected/35a14012-73eb-4f3b-b762-f56c6a963e44-kube-api-access-sgkdb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:46:51.503733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-trusted-ca\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.503733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503539 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:46:51.503733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64l24\" (UniqueName: \"kubernetes.io/projected/3cdefc81-0135-4053-bc7b-78958686e8ea-kube-api-access-64l24\") pod \"network-check-source-6f78d59f99-jdsr5\" (UID: \"3cdefc81-0135-4053-bc7b-78958686e8ea\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5" Oct 30 16:46:51.503733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503572 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f13fd623-29b9-4d8b-b76d-668ca123010f-tmp-dir\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:51.503733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503589 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-config\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:46:51.503733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503629 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dacad254-b0c1-48fd-8377-1067d65ea230-config\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.503733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503659 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-serving-cert\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:46:51.503733 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.503672 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:46:51.503733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:46:51.503733 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.503728 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls podName:b34094c9-7349-4284-8b86-2253f9e4d87f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:52.003713302 +0000 UTC m=+30.685916482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fpc52" (UID: "b34094c9-7349-4284-8b86-2253f9e4d87f") : secret "samples-operator-tls" not found Oct 30 16:46:51.504376 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.503756 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:46:51.504376 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.503830 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls podName:dbb5e2c6-44f4-4152-a7ac-98ae4c045190 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:52.00381476 +0000 UTC m=+30.686017932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c69x2" (UID: "dbb5e2c6-44f4-4152-a7ac-98ae4c045190") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:46:51.504376 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-installation-pull-secrets\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.504376 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503861 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.504376 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503930 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f13fd623-29b9-4d8b-b76d-668ca123010f-config-volume\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:51.504376 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.503952 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:46:51.504376 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.503965 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67785c878-8c9pr: secret "image-registry-tls" not found Oct 30 16:46:51.504376 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.503962 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3794a50e-6c2f-40bc-9be7-eda1388f3178-ca-trust-extracted\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.504376 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.504000 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls podName:3794a50e-6c2f-40bc-9be7-eda1388f3178 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:52.003987564 +0000 UTC m=+30.686190753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls") pod "image-registry-67785c878-8c9pr" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178") : secret "image-registry-tls" not found Oct 30 16:46:51.504376 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504267 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-certificates\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b516f174-836e-4711-8999-80ce5a592e28-serving-cert\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504479 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dacad254-b0c1-48fd-8377-1067d65ea230-trusted-ca\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504527 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b516f174-836e-4711-8999-80ce5a592e28-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dacad254-b0c1-48fd-8377-1067d65ea230-serving-cert\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504560 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dacad254-b0c1-48fd-8377-1067d65ea230-config\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504582 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504614 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-image-registry-private-configuration\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504645 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmgvx\" (UniqueName: \"kubernetes.io/projected/b34094c9-7349-4284-8b86-2253f9e4d87f-kube-api-access-zmgvx\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504682 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35a14012-73eb-4f3b-b762-f56c6a963e44-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b516f174-836e-4711-8999-80ce5a592e28-tmp\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504733 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504757 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504787 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm5w2\" (UniqueName: \"kubernetes.io/projected/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-kube-api-access-gm5w2\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504824 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b516f174-836e-4711-8999-80ce5a592e28-service-ca-bundle\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.504884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504877 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqff6\" (UniqueName: \"kubernetes.io/projected/f13fd623-29b9-4d8b-b76d-668ca123010f-kube-api-access-sqff6\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:51.505617 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8f6w\" (UniqueName: \"kubernetes.io/projected/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-kube-api-access-z8f6w\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:46:51.505617 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504942 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b516f174-836e-4711-8999-80ce5a592e28-snapshots\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.505617 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504998 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-bound-sa-token\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.505617 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.505032 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35a14012-73eb-4f3b-b762-f56c6a963e44-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:46:51.505617 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.505062 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:46:51.505617 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.505087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6zs9\" (UniqueName: \"kubernetes.io/projected/b516f174-836e-4711-8999-80ce5a592e28-kube-api-access-q6zs9\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.505617 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.505110 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68321123-f8fc-4c6a-bd22-1f614a8a3b55-cert\") pod \"ingress-canary-tb2dr\" (UID: \"68321123-f8fc-4c6a-bd22-1f614a8a3b55\") " pod="openshift-ingress-canary/ingress-canary-tb2dr" Oct 30 16:46:51.505617 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.505107 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-trusted-ca\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.505617 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.505136 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:46:51.505617 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.505197 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls podName:f13fd623-29b9-4d8b-b76d-668ca123010f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:52.005181969 +0000 UTC m=+30.687385130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls") pod "dns-default-6rrfj" (UID: "f13fd623-29b9-4d8b-b76d-668ca123010f") : secret "dns-default-metrics-tls" not found Oct 30 16:46:51.505617 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.505395 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b516f174-836e-4711-8999-80ce5a592e28-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.506153 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.505807 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35a14012-73eb-4f3b-b762-f56c6a963e44-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:46:51.506153 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.504564 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-config\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:46:51.506153 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.505933 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:46:51.506153 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.506054 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b516f174-836e-4711-8999-80ce5a592e28-service-ca-bundle\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.506461 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.506436 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:46:51.506685 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.506659 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-serving-cert\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:46:51.506782 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.506758 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-installation-pull-secrets\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.506880 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.506856 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b516f174-836e-4711-8999-80ce5a592e28-tmp\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.506996 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.506976 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b516f174-836e-4711-8999-80ce5a592e28-snapshots\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.507103 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.506986 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:46:51.507224 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.507205 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dacad254-b0c1-48fd-8377-1067d65ea230-serving-cert\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.508389 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.508361 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35a14012-73eb-4f3b-b762-f56c6a963e44-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:46:51.508389 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.508370 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b516f174-836e-4711-8999-80ce5a592e28-serving-cert\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.508737 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.508718 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68321123-f8fc-4c6a-bd22-1f614a8a3b55-cert\") pod \"ingress-canary-tb2dr\" (UID: \"68321123-f8fc-4c6a-bd22-1f614a8a3b55\") " pod="openshift-ingress-canary/ingress-canary-tb2dr" Oct 30 16:46:51.509015 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.508999 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-image-registry-private-configuration\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.528997 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.528972 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cmkh\" (UniqueName: \"kubernetes.io/projected/dacad254-b0c1-48fd-8377-1067d65ea230-kube-api-access-8cmkh\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.530800 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.530784 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fb8qf" Oct 30 16:46:51.535500 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.535476 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqff6\" (UniqueName: \"kubernetes.io/projected/f13fd623-29b9-4d8b-b76d-668ca123010f-kube-api-access-sqff6\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:51.535643 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.535626 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qprsq\" (UniqueName: \"kubernetes.io/projected/68321123-f8fc-4c6a-bd22-1f614a8a3b55-kube-api-access-qprsq\") pod \"ingress-canary-tb2dr\" (UID: \"68321123-f8fc-4c6a-bd22-1f614a8a3b55\") " pod="openshift-ingress-canary/ingress-canary-tb2dr" Oct 30 16:46:51.538083 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.538060 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mmhr5" Oct 30 16:46:51.538310 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.538290 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-bound-sa-token\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.539037 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.538999 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgkdb\" (UniqueName: \"kubernetes.io/projected/35a14012-73eb-4f3b-b762-f56c6a963e44-kube-api-access-sgkdb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:46:51.539189 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.539167 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8f6w\" (UniqueName: \"kubernetes.io/projected/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-kube-api-access-z8f6w\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:46:51.539269 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.539248 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm5w2\" (UniqueName: \"kubernetes.io/projected/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-kube-api-access-gm5w2\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:46:51.539391 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.539367 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64l24\" (UniqueName: \"kubernetes.io/projected/3cdefc81-0135-4053-bc7b-78958686e8ea-kube-api-access-64l24\") pod \"network-check-source-6f78d59f99-jdsr5\" (UID: \"3cdefc81-0135-4053-bc7b-78958686e8ea\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5" Oct 30 16:46:51.539509 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.539488 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xjxv\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-kube-api-access-7xjxv\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:51.540690 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.540665 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmgvx\" (UniqueName: \"kubernetes.io/projected/b34094c9-7349-4284-8b86-2253f9e4d87f-kube-api-access-zmgvx\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:46:51.541042 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.541027 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6zs9\" (UniqueName: \"kubernetes.io/projected/b516f174-836e-4711-8999-80ce5a592e28-kube-api-access-q6zs9\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.551434 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.551416 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mhsj5" Oct 30 16:46:51.559180 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.559164 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56" Oct 30 16:46:51.578823 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.578803 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:46:51.602121 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.602076 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:46:51.608183 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.608083 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:46:51.677204 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.677182 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:46:51.702928 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.695699 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5" Oct 30 16:46:51.747598 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.747414 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tb2dr" Oct 30 16:46:51.910871 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.910784 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.911310 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.910939 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:51.911310 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.910976 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:52.910952966 +0000 UTC m=+31.593156139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : configmap references non-existent config key: service-ca.crt Oct 30 16:46:51.911310 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.911026 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:51.911310 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.911046 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:46:51.911310 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.911061 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d7cb6cf-rzmrh: secret "image-registry-tls" not found Oct 30 16:46:51.911310 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.911108 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls podName:d38312b4-bd6e-4a5f-8a33-d93fb827debc nodeName:}" failed. No retries permitted until 2026-10-30 16:46:52.911093661 +0000 UTC m=+31.593296819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls") pod "image-registry-797d7cb6cf-rzmrh" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc") : secret "image-registry-tls" not found Oct 30 16:46:51.911310 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.911155 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:46:51.911310 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:51.911212 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:52.911196544 +0000 UTC m=+31.593399719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : secret "router-metrics-certs-default" not found Oct 30 16:46:51.919140 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.919119 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mmhr5"] Oct 30 16:46:51.919542 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:51.919509 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30001919_517f_4031_9ef4_bc0fbb43d122.slice/crio-ecbf1d255616640b88798eb88bb8b0b048bfc7bc5cca973fa973d826ee4f92e8 WatchSource:0}: Error finding container ecbf1d255616640b88798eb88bb8b0b048bfc7bc5cca973fa973d826ee4f92e8: Status 404 returned error can't find the container with id ecbf1d255616640b88798eb88bb8b0b048bfc7bc5cca973fa973d826ee4f92e8 Oct 30 16:46:51.921884 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.921863 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fb8qf"] Oct 30 16:46:51.922180 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:51.922158 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3095c033_c4ff_42cd_9530_1be17a144a31.slice/crio-1e59114d4ba6e950fd87cd095372196c103e71fa14acf4b788c9d916fe99b0a4 WatchSource:0}: Error finding container 1e59114d4ba6e950fd87cd095372196c103e71fa14acf4b788c9d916fe99b0a4: Status 404 returned error can't find the container with id 1e59114d4ba6e950fd87cd095372196c103e71fa14acf4b788c9d916fe99b0a4 Oct 30 16:46:51.933968 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.933719 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45"] Oct 30 16:46:51.934516 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:51.934487 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11fe6a2d_ed34_40cb_9a73_a637bd3a2d6e.slice/crio-03ebb684cd0cb48ec84a17364457a71e57eba2689747985e90df2b2ed4da1303 WatchSource:0}: Error finding container 03ebb684cd0cb48ec84a17364457a71e57eba2689747985e90df2b2ed4da1303: Status 404 returned error can't find the container with id 03ebb684cd0cb48ec84a17364457a71e57eba2689747985e90df2b2ed4da1303 Oct 30 16:46:51.953507 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.953473 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mhsj5"] Oct 30 16:46:51.995657 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:51.995624 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56"] Oct 30 16:46:51.999997 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:51.999861 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a04182_8031_4fa8_8bed_4f8b60b6cc97.slice/crio-1da9fa642e9eaa3fbf7233d736ecacb10e621a2130b022ea6b1b02ab1aa1f2f0 WatchSource:0}: Error finding container 1da9fa642e9eaa3fbf7233d736ecacb10e621a2130b022ea6b1b02ab1aa1f2f0: Status 404 returned error can't find the container with id 1da9fa642e9eaa3fbf7233d736ecacb10e621a2130b022ea6b1b02ab1aa1f2f0 Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.012306 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.012390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.012437 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.012467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.012526 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.012661 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.012730 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls podName:f13fd623-29b9-4d8b-b76d-668ca123010f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:53.012712665 +0000 UTC m=+31.694915825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls") pod "dns-default-6rrfj" (UID: "f13fd623-29b9-4d8b-b76d-668ca123010f") : secret "dns-default-metrics-tls" not found Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.012793 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.012826 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert podName:7918c4b3-b4f4-4435-9de9-77fbe4906a7f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:53.012815638 +0000 UTC m=+31.695018798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7zrs4" (UID: "7918c4b3-b4f4-4435-9de9-77fbe4906a7f") : secret "networking-console-plugin-cert" not found Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.012878 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.012925 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls podName:b34094c9-7349-4284-8b86-2253f9e4d87f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:53.012914656 +0000 UTC m=+31.695117819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fpc52" (UID: "b34094c9-7349-4284-8b86-2253f9e4d87f") : secret "samples-operator-tls" not found Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.012978 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.013009 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls podName:dbb5e2c6-44f4-4152-a7ac-98ae4c045190 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:53.012999686 +0000 UTC m=+31.695202845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c69x2" (UID: "dbb5e2c6-44f4-4152-a7ac-98ae4c045190") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.013066 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.013078 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67785c878-8c9pr: secret "image-registry-tls" not found Oct 30 16:46:52.015188 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.013107 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls podName:3794a50e-6c2f-40bc-9be7-eda1388f3178 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:53.013097907 +0000 UTC m=+31.695301067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls") pod "image-registry-67785c878-8c9pr" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178") : secret "image-registry-tls" not found Oct 30 16:46:52.016190 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.015083 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq"] Oct 30 16:46:52.017454 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:52.017363 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a14012_73eb_4f3b_b762_f56c6a963e44.slice/crio-a3cc7e5082a0f483f87a3aff41083d0f9e6f8989b11dcb35e38ef070a1f7bb0a WatchSource:0}: Error finding container a3cc7e5082a0f483f87a3aff41083d0f9e6f8989b11dcb35e38ef070a1f7bb0a: Status 404 returned error can't find the container with id a3cc7e5082a0f483f87a3aff41083d0f9e6f8989b11dcb35e38ef070a1f7bb0a Oct 30 16:46:52.042305 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.042273 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jmx89"] Oct 30 16:46:52.043102 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:52.043076 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb516f174_836e_4711_8999_80ce5a592e28.slice/crio-f18612a92e2dc29542be8b2489f9ff7c492ee84b563208eb8fbfba4460c0aecc WatchSource:0}: Error finding container f18612a92e2dc29542be8b2489f9ff7c492ee84b563208eb8fbfba4460c0aecc: Status 404 returned error can't find the container with id f18612a92e2dc29542be8b2489f9ff7c492ee84b563208eb8fbfba4460c0aecc Oct 30 16:46:52.063026 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.062999 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mhsj5" event={"ID":"159cde9b-22a5-4930-9bf6-2535ab4af3d4","Type":"ContainerStarted","Data":"a196d593fd53d6c5c9b76e4ba832465346894a7262a169b1b18c344b594b65c9"} Oct 30 16:46:52.063990 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.063966 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" event={"ID":"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e","Type":"ContainerStarted","Data":"03ebb684cd0cb48ec84a17364457a71e57eba2689747985e90df2b2ed4da1303"} Oct 30 16:46:52.065709 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.065686 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fb8qf" event={"ID":"3095c033-c4ff-42cd-9530-1be17a144a31","Type":"ContainerStarted","Data":"1e59114d4ba6e950fd87cd095372196c103e71fa14acf4b788c9d916fe99b0a4"} Oct 30 16:46:52.066885 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.066865 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56" event={"ID":"d7a04182-8031-4fa8-8bed-4f8b60b6cc97","Type":"ContainerStarted","Data":"1da9fa642e9eaa3fbf7233d736ecacb10e621a2130b022ea6b1b02ab1aa1f2f0"} Oct 30 16:46:52.068454 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.068406 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mmhr5" event={"ID":"30001919-517f-4031-9ef4-bc0fbb43d122","Type":"ContainerStarted","Data":"ecbf1d255616640b88798eb88bb8b0b048bfc7bc5cca973fa973d826ee4f92e8"} Oct 30 16:46:52.069455 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.069405 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" event={"ID":"35a14012-73eb-4f3b-b762-f56c6a963e44","Type":"ContainerStarted","Data":"a3cc7e5082a0f483f87a3aff41083d0f9e6f8989b11dcb35e38ef070a1f7bb0a"} Oct 30 16:46:52.070799 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.070780 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jmx89" event={"ID":"b516f174-836e-4711-8999-80ce5a592e28","Type":"ContainerStarted","Data":"f18612a92e2dc29542be8b2489f9ff7c492ee84b563208eb8fbfba4460c0aecc"} Oct 30 16:46:52.075013 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.074989 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-n758f"] Oct 30 16:46:52.076040 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:52.076007 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacad254_b0c1_48fd_8377_1067d65ea230.slice/crio-774df0d8e22e6bacf27357b34b44bd971a3a2680b7bcb2bfd028b632d778ea35 WatchSource:0}: Error finding container 774df0d8e22e6bacf27357b34b44bd971a3a2680b7bcb2bfd028b632d778ea35: Status 404 returned error can't find the container with id 774df0d8e22e6bacf27357b34b44bd971a3a2680b7bcb2bfd028b632d778ea35 Oct 30 16:46:52.077961 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.077942 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5"] Oct 30 16:46:52.078440 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:52.078422 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdefc81_0135_4053_bc7b_78958686e8ea.slice/crio-0b166ba605a74b057cd8ab0032f9e35754ce05745876024b528a0b101fba0e4c WatchSource:0}: Error finding container 0b166ba605a74b057cd8ab0032f9e35754ce05745876024b528a0b101fba0e4c: Status 404 returned error can't find the container with id 0b166ba605a74b057cd8ab0032f9e35754ce05745876024b528a0b101fba0e4c Oct 30 16:46:52.111507 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:52.111467 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68321123_f8fc_4c6a_bd22_1f614a8a3b55.slice/crio-77affc7483735aab7d4b37a8768bcd98379bf5cc9f4d73e893cf3e1fd40f69bb WatchSource:0}: Error finding container 77affc7483735aab7d4b37a8768bcd98379bf5cc9f4d73e893cf3e1fd40f69bb: Status 404 returned error can't find the container with id 77affc7483735aab7d4b37a8768bcd98379bf5cc9f4d73e893cf3e1fd40f69bb Oct 30 16:46:52.112126 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.112101 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tb2dr"] Oct 30 16:46:52.797629 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.797593 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:52.797858 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.797632 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:52.797858 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.797822 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:52.814914 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.814852 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:46:52.815733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.815284 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bdrw\"" Oct 30 16:46:52.815733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.815335 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szhrj\"" Oct 30 16:46:52.815733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.815544 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:46:52.815733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.815672 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8568q\"" Oct 30 16:46:52.920001 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.919966 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:52.920458 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.920016 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:52.920458 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:52.920143 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:52.920458 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.920356 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:54.920336782 +0000 UTC m=+33.602539953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : configmap references non-existent config key: service-ca.crt Oct 30 16:46:52.920793 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.920771 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:46:52.920865 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.920797 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d7cb6cf-rzmrh: secret "image-registry-tls" not found Oct 30 16:46:52.920865 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.920844 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls podName:d38312b4-bd6e-4a5f-8a33-d93fb827debc nodeName:}" failed. No retries permitted until 2026-10-30 16:46:54.920828804 +0000 UTC m=+33.603031976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls") pod "image-registry-797d7cb6cf-rzmrh" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc") : secret "image-registry-tls" not found Oct 30 16:46:52.920988 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.920932 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:46:52.920988 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:52.920969 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:54.920957163 +0000 UTC m=+33.603160346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : secret "router-metrics-certs-default" not found Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.020875 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.021023 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:53.021183 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:53.021344 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.021413 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.021459 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.021492 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:53.021587 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:53.021599 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67785c878-8c9pr: secret "image-registry-tls" not found Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:53.021640 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls podName:3794a50e-6c2f-40bc-9be7-eda1388f3178 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:55.021621792 +0000 UTC m=+33.703824963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls") pod "image-registry-67785c878-8c9pr" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178") : secret "image-registry-tls" not found Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:53.021667 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert podName:7918c4b3-b4f4-4435-9de9-77fbe4906a7f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:55.021657209 +0000 UTC m=+33.703860374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7zrs4" (UID: "7918c4b3-b4f4-4435-9de9-77fbe4906a7f") : secret "networking-console-plugin-cert" not found Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:53.021684 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls podName:f13fd623-29b9-4d8b-b76d-668ca123010f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:55.021675575 +0000 UTC m=+33.703878743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls") pod "dns-default-6rrfj" (UID: "f13fd623-29b9-4d8b-b76d-668ca123010f") : secret "dns-default-metrics-tls" not found Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:53.021742 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:53.021833 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls podName:b34094c9-7349-4284-8b86-2253f9e4d87f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:55.021822182 +0000 UTC m=+33.704025354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fpc52" (UID: "b34094c9-7349-4284-8b86-2253f9e4d87f") : secret "samples-operator-tls" not found Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:53.021971 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:46:53.022354 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:53.022028 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls podName:dbb5e2c6-44f4-4152-a7ac-98ae4c045190 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:55.022013616 +0000 UTC m=+33.704216788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c69x2" (UID: "dbb5e2c6-44f4-4152-a7ac-98ae4c045190") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:46:53.081101 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.080953 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tb2dr" event={"ID":"68321123-f8fc-4c6a-bd22-1f614a8a3b55","Type":"ContainerStarted","Data":"77affc7483735aab7d4b37a8768bcd98379bf5cc9f4d73e893cf3e1fd40f69bb"} Oct 30 16:46:53.085420 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.085388 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mmhr5" event={"ID":"30001919-517f-4031-9ef4-bc0fbb43d122","Type":"ContainerStarted","Data":"72a56127947bd8ba46b2b7c045d19dc594182ecc7f191782342713071ed2f20f"} Oct 30 16:46:53.088661 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.088609 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5" event={"ID":"3cdefc81-0135-4053-bc7b-78958686e8ea","Type":"ContainerStarted","Data":"0b166ba605a74b057cd8ab0032f9e35754ce05745876024b528a0b101fba0e4c"} Oct 30 16:46:53.090296 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.090271 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" event={"ID":"dacad254-b0c1-48fd-8377-1067d65ea230","Type":"ContainerStarted","Data":"774df0d8e22e6bacf27357b34b44bd971a3a2680b7bcb2bfd028b632d778ea35"} Oct 30 16:46:53.094796 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.094771 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mhsj5" event={"ID":"159cde9b-22a5-4930-9bf6-2535ab4af3d4","Type":"ContainerStarted","Data":"f2f49b228f91e0367769151eac8f856944dacf80ed84b61cbdef3367ee096ae3"} Oct 30 16:46:53.098786 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.098327 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fb8qf" event={"ID":"3095c033-c4ff-42cd-9530-1be17a144a31","Type":"ContainerStarted","Data":"0e10542f9a634bb69ca1e55cecda8191c0c14f03979b19dd1865ec3caad62499"} Oct 30 16:46:53.129412 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.129353 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-mmhr5" podStartSLOduration=392.129305546 podStartE2EDuration="6m32.129305546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:53.128635375 +0000 UTC m=+31.810838557" watchObservedRunningTime="2026-10-30 16:46:53.129305546 +0000 UTC m=+31.811508722" Oct 30 16:46:53.172951 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.172869 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fb8qf" podStartSLOduration=392.172850526 podStartE2EDuration="6m32.172850526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:53.172049391 +0000 UTC m=+31.854252573" watchObservedRunningTime="2026-10-30 16:46:53.172850526 +0000 UTC m=+31.855053706" Oct 30 16:46:53.221951 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:53.221879 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-mhsj5" podStartSLOduration=392.221861197 podStartE2EDuration="6m32.221861197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:53.219468707 +0000 UTC m=+31.901671888" watchObservedRunningTime="2026-10-30 16:46:53.221861197 +0000 UTC m=+31.904064380" Oct 30 16:46:54.106841 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:54.106695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:54.317856 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:54.317823 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:54.945126 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:54.945082 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:54.945329 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:54.945240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:54.945329 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:54.945269 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:54.945417 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:54.945395 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:46:54.945452 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:54.945443 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:58.94543054 +0000 UTC m=+37.627633712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : secret "router-metrics-certs-default" not found Oct 30 16:46:54.945817 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:54.945802 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:58.945789917 +0000 UTC m=+37.627993088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : configmap references non-existent config key: service-ca.crt Oct 30 16:46:54.945936 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:54.945867 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:46:54.945936 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:54.945878 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d7cb6cf-rzmrh: secret "image-registry-tls" not found Oct 30 16:46:54.945936 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:54.945914 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls podName:d38312b4-bd6e-4a5f-8a33-d93fb827debc nodeName:}" failed. No retries permitted until 2026-10-30 16:46:58.945892972 +0000 UTC m=+37.628096131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls") pod "image-registry-797d7cb6cf-rzmrh" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc") : secret "image-registry-tls" not found Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.046426 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.046469 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.046532 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.046616 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:55.046620 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.046660 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:55.046692 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls podName:dbb5e2c6-44f4-4152-a7ac-98ae4c045190 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.046673523 +0000 UTC m=+37.728876682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c69x2" (UID: "dbb5e2c6-44f4-4152-a7ac-98ae4c045190") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:55.046759 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:55.046804 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls podName:b34094c9-7349-4284-8b86-2253f9e4d87f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.046792814 +0000 UTC m=+37.728995976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fpc52" (UID: "b34094c9-7349-4284-8b86-2253f9e4d87f") : secret "samples-operator-tls" not found Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:55.046852 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:55.046860 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67785c878-8c9pr: secret "image-registry-tls" not found Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:55.046883 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls podName:3794a50e-6c2f-40bc-9be7-eda1388f3178 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.04687403 +0000 UTC m=+37.729077188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls") pod "image-registry-67785c878-8c9pr" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178") : secret "image-registry-tls" not found Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:55.046937 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:55.046960 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls podName:f13fd623-29b9-4d8b-b76d-668ca123010f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.04695136 +0000 UTC m=+37.729154518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls") pod "dns-default-6rrfj" (UID: "f13fd623-29b9-4d8b-b76d-668ca123010f") : secret "dns-default-metrics-tls" not found Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:55.046995 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:46:55.052919 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:55.047015 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert podName:7918c4b3-b4f4-4435-9de9-77fbe4906a7f nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.047009854 +0000 UTC m=+37.729213012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7zrs4" (UID: "7918c4b3-b4f4-4435-9de9-77fbe4906a7f") : secret "networking-console-plugin-cert" not found Oct 30 16:46:55.111527 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.110565 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56" event={"ID":"d7a04182-8031-4fa8-8bed-4f8b60b6cc97","Type":"ContainerStarted","Data":"36ead6b665d14207f2da87d3d35a9ff51e783a53f6149d9454ae8de2352c17c5"} Oct 30 16:46:55.153792 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.153731 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56" podStartSLOduration=360.650493627 podStartE2EDuration="6m3.153717052s" totalImagesPullingTime="2.503223425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:55.153678533 +0000 UTC m=+33.835881713" watchObservedRunningTime="2026-10-30 16:46:55.153717052 +0000 UTC m=+33.835920233" Oct 30 16:46:55.326927 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.326879 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:55.484187 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.484102 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:55.553064 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.553027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:55.553215 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.553113 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:46:55.553260 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:55.553228 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:46:55.553293 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:55.553272 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs podName:3ca4c236-e28d-4331-9336-63a71d706ef0 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:27.553260797 +0000 UTC m=+66.235463955 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs") pod "network-metrics-daemon-8mpgm" (UID: "3ca4c236-e28d-4331-9336-63a71d706ef0") : secret "metrics-daemon-secret" not found Oct 30 16:46:55.556481 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.556421 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:55.574587 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.574556 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:46:55.654644 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.654602 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzz5\" (UniqueName: \"kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5\") pod \"network-check-target-scdcl\" (UID: \"a132686f-9475-437d-826e-22fb93e86323\") " pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:55.661029 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.661002 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzz5\" (UniqueName: \"kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5\") pod \"network-check-target-scdcl\" (UID: \"a132686f-9475-437d-826e-22fb93e86323\") " pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:55.824661 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.824625 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:46:55.913282 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.913196 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wgxv6"] Oct 30 16:46:55.980338 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:55.980304 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:56.115606 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:56.115529 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wgxv6" event={"ID":"ac8ed89d-9808-4bff-9d24-3e5fc34041ad","Type":"ContainerStarted","Data":"b3c7cd045ffa4e727bde3156153e89aa8677f1e94e56a90ceec2ba11980727bf"} Oct 30 16:46:56.152147 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:56.152098 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:56.163699 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:56.163670 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-scdcl"] Oct 30 16:46:56.170440 ip-10-0-135-153 kubenswrapper[2562]: W1030 16:46:56.170410 2562 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda132686f_9475_437d_826e_22fb93e86323.slice/crio-dbe6588c13763b48fd28430b4f4c5429731f4c47e52fee7e9a79177f59fcfbd0 WatchSource:0}: Error finding container dbe6588c13763b48fd28430b4f4c5429731f4c47e52fee7e9a79177f59fcfbd0: Status 404 returned error can't find the container with id dbe6588c13763b48fd28430b4f4c5429731f4c47e52fee7e9a79177f59fcfbd0 Oct 30 16:46:57.120157 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:57.120127 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/0.log" Oct 30 16:46:57.120623 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:57.120179 2562 generic.go:356] "Generic (PLEG): container finished" podID="dacad254-b0c1-48fd-8377-1067d65ea230" containerID="05b2908701dbc77d4b29afc47ebed2610abedca26c914f87a9fa5bfed9d3c34d" exitCode=255 Oct 30 16:46:57.120623 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:57.120272 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" event={"ID":"dacad254-b0c1-48fd-8377-1067d65ea230","Type":"ContainerDied","Data":"05b2908701dbc77d4b29afc47ebed2610abedca26c914f87a9fa5bfed9d3c34d"} Oct 30 16:46:57.120623 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:57.120543 2562 scope.go:122] "RemoveContainer" containerID="05b2908701dbc77d4b29afc47ebed2610abedca26c914f87a9fa5bfed9d3c34d" Oct 30 16:46:57.121948 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:57.121928 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-scdcl" event={"ID":"a132686f-9475-437d-826e-22fb93e86323","Type":"ContainerStarted","Data":"dbe6588c13763b48fd28430b4f4c5429731f4c47e52fee7e9a79177f59fcfbd0"} Oct 30 16:46:58.128096 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.128057 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" event={"ID":"35a14012-73eb-4f3b-b762-f56c6a963e44","Type":"ContainerStarted","Data":"071604208a7d920227d29eaa1f76504d152027748a890939553f726b9fae546a"} Oct 30 16:46:58.130539 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.130515 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jmx89" event={"ID":"b516f174-836e-4711-8999-80ce5a592e28","Type":"ContainerStarted","Data":"4c74f557998eabc62244dafb8eefeb0215c38c04a62d9f7eadcd7d8adcc70519"} Oct 30 16:46:58.132364 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.132338 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" event={"ID":"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e","Type":"ContainerStarted","Data":"0ea948261e59c5dc96e8264981118d5b3947060bc0a50af3b0f5c640645c1df0"} Oct 30 16:46:58.134549 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.134489 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tb2dr" event={"ID":"68321123-f8fc-4c6a-bd22-1f614a8a3b55","Type":"ContainerStarted","Data":"39662fca3a0fee677860bffef9ceff40494477fdd93d311e59a44bd8ae6e5f62"} Oct 30 16:46:58.138483 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.138266 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5" event={"ID":"3cdefc81-0135-4053-bc7b-78958686e8ea","Type":"ContainerStarted","Data":"ebae50d2b220ab75d92cf6e183d682a35024a9f40f554151f0a4dfcf4fe3af06"} Oct 30 16:46:58.143208 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.143186 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/1.log" Oct 30 16:46:58.143697 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.143681 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/0.log" Oct 30 16:46:58.143798 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.143717 2562 generic.go:356] "Generic (PLEG): container finished" podID="dacad254-b0c1-48fd-8377-1067d65ea230" containerID="c99e50eea3562218c0bb5c17d257f8fbe287413615ed42c5c59b5e6a7a33f71f" exitCode=255 Oct 30 16:46:58.143798 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.143750 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" event={"ID":"dacad254-b0c1-48fd-8377-1067d65ea230","Type":"ContainerDied","Data":"c99e50eea3562218c0bb5c17d257f8fbe287413615ed42c5c59b5e6a7a33f71f"} Oct 30 16:46:58.143798 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.143791 2562 scope.go:122] "RemoveContainer" containerID="05b2908701dbc77d4b29afc47ebed2610abedca26c914f87a9fa5bfed9d3c34d" Oct 30 16:46:58.144158 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.144142 2562 scope.go:122] "RemoveContainer" containerID="c99e50eea3562218c0bb5c17d257f8fbe287413615ed42c5c59b5e6a7a33f71f" Oct 30 16:46:58.144327 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:58.144307 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n758f_openshift-console-operator(dacad254-b0c1-48fd-8377-1067d65ea230)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" podUID="dacad254-b0c1-48fd-8377-1067d65ea230" Oct 30 16:46:58.176929 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.173743 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" podStartSLOduration=360.78102466 podStartE2EDuration="6m6.173727868s" totalImagesPullingTime="5.392703208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:58.173107134 +0000 UTC m=+36.855310320" watchObservedRunningTime="2026-10-30 16:46:58.173727868 +0000 UTC m=+36.855931047" Oct 30 16:46:58.226126 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.226067 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-jmx89" podStartSLOduration=360.863436354 podStartE2EDuration="6m6.226050062s" totalImagesPullingTime="5.362613708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:58.225992331 +0000 UTC m=+36.908195508" watchObservedRunningTime="2026-10-30 16:46:58.226050062 +0000 UTC m=+36.908253242" Oct 30 16:46:58.306091 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.305488 2562 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j7sxm"] Oct 30 16:46:58.328534 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.328501 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.330837 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.330792 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tb2dr" podStartSLOduration=1.9967715799999999 podStartE2EDuration="7.330778866s" totalImagesPullingTime="5.334007286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:58.329797575 +0000 UTC m=+37.012000755" watchObservedRunningTime="2026-10-30 16:46:58.330778866 +0000 UTC m=+37.012982043" Oct 30 16:46:58.358142 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.358107 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:46:58.358424 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.358240 2562 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7stlv\"" Oct 30 16:46:58.364855 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.364776 2562 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7sxm"] Oct 30 16:46:58.369384 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.369356 2562 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:46:58.402955 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.402861 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" podStartSLOduration=360.922472639 podStartE2EDuration="6m6.402844186s" totalImagesPullingTime="5.480371547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:58.400683473 +0000 UTC m=+37.082886655" watchObservedRunningTime="2026-10-30 16:46:58.402844186 +0000 UTC m=+37.085047366" Oct 30 16:46:58.439303 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.439210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:58.459165 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.459100 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5" podStartSLOduration=359.715739969 podStartE2EDuration="6m5.459079453s" totalImagesPullingTime="5.743339484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:58.456486655 +0000 UTC m=+37.138689837" watchObservedRunningTime="2026-10-30 16:46:58.459079453 +0000 UTC m=+37.141282634" Oct 30 16:46:58.486894 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.486859 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.487067 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.486939 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/883b14e2-b8bc-4650-a638-c33673683a45-data-volume\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.487067 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.486984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.487067 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.487052 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwc77\" (UniqueName: \"kubernetes.io/projected/883b14e2-b8bc-4650-a638-c33673683a45-kube-api-access-fwc77\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.487224 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.487088 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/883b14e2-b8bc-4650-a638-c33673683a45-crio-socket\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.588554 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.588508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.588723 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:58.588626 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:46:58.588723 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.588636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwc77\" (UniqueName: \"kubernetes.io/projected/883b14e2-b8bc-4650-a638-c33673683a45-kube-api-access-fwc77\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.588723 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.588681 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/883b14e2-b8bc-4650-a638-c33673683a45-crio-socket\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.588871 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.588744 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/883b14e2-b8bc-4650-a638-c33673683a45-crio-socket\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.588871 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:58.588783 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.088675209 +0000 UTC m=+37.770878391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : configmap "kube-rbac-proxy" not found Oct 30 16:46:58.589019 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.588923 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.589019 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.588968 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/883b14e2-b8bc-4650-a638-c33673683a45-data-volume\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.589118 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:58.589049 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:46:58.589118 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:58.589092 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:46:59.089080752 +0000 UTC m=+37.771283926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : secret "insights-runtime-extractor-tls" not found Oct 30 16:46:58.589357 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.589330 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/883b14e2-b8bc-4650-a638-c33673683a45-data-volume\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.622479 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.622431 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:58.629847 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.629788 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwc77\" (UniqueName: \"kubernetes.io/projected/883b14e2-b8bc-4650-a638-c33673683a45-kube-api-access-fwc77\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:58.803262 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.803216 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:58.960946 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.960890 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:58.993147 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.993109 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:58.993323 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.993230 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:46:58.993323 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:58.993266 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:46:58.993430 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:58.993345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:46:58.993470 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:58.993443 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:46:58.993470 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:58.993455 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d7cb6cf-rzmrh: secret "image-registry-tls" not found Oct 30 16:46:58.993528 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:58.993500 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls podName:d38312b4-bd6e-4a5f-8a33-d93fb827debc nodeName:}" failed. No retries permitted until 2026-10-30 16:47:06.993481923 +0000 UTC m=+45.675685094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls") pod "image-registry-797d7cb6cf-rzmrh" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc") : secret "image-registry-tls" not found Oct 30 16:46:58.993582 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:58.993563 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:06.993552284 +0000 UTC m=+45.675755459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : configmap references non-existent config key: service-ca.crt Oct 30 16:46:58.993582 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:58.993587 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:06.993579992 +0000 UTC m=+45.675783150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : secret "router-metrics-certs-default" not found Oct 30 16:46:59.094795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:59.094718 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:46:59.094795 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:59.094775 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:59.095053 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:59.094853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:46:59.095053 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.094877 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:46:59.095053 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:59.094914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:46:59.095053 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.094956 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls podName:f13fd623-29b9-4d8b-b76d-668ca123010f nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.094937276 +0000 UTC m=+45.777140447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls") pod "dns-default-6rrfj" (UID: "f13fd623-29b9-4d8b-b76d-668ca123010f") : secret "dns-default-metrics-tls" not found Oct 30 16:46:59.095053 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:59.095006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:46:59.095053 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.095018 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:46:59.095053 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:59.095032 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:46:59.095053 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:59.095057 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:46:59.095447 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.095072 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert podName:7918c4b3-b4f4-4435-9de9-77fbe4906a7f nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.095057071 +0000 UTC m=+45.777260230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7zrs4" (UID: "7918c4b3-b4f4-4435-9de9-77fbe4906a7f") : secret "networking-console-plugin-cert" not found Oct 30 16:46:59.095447 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.095137 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:46:59.095447 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.095171 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:00.095160421 +0000 UTC m=+38.777363582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : secret "insights-runtime-extractor-tls" not found Oct 30 16:46:59.095447 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.095214 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:46:59.095447 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.095247 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:00.09523643 +0000 UTC m=+38.777439590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : configmap "kube-rbac-proxy" not found Oct 30 16:46:59.095447 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.095290 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:46:59.095447 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.095301 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67785c878-8c9pr: secret "image-registry-tls" not found Oct 30 16:46:59.095447 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.095345 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls podName:3794a50e-6c2f-40bc-9be7-eda1388f3178 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.095334917 +0000 UTC m=+45.777538090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls") pod "image-registry-67785c878-8c9pr" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178") : secret "image-registry-tls" not found Oct 30 16:46:59.095447 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.095370 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:46:59.095447 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.095402 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls podName:b34094c9-7349-4284-8b86-2253f9e4d87f nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.095393159 +0000 UTC m=+45.777596338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fpc52" (UID: "b34094c9-7349-4284-8b86-2253f9e4d87f") : secret "samples-operator-tls" not found Oct 30 16:46:59.095447 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.095406 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:46:59.095447 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.095442 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls podName:dbb5e2c6-44f4-4152-a7ac-98ae4c045190 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.09543038 +0000 UTC m=+45.777633554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c69x2" (UID: "dbb5e2c6-44f4-4152-a7ac-98ae4c045190") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:46:59.148734 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:59.148704 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/1.log" Oct 30 16:46:59.149204 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:59.149113 2562 scope.go:122] "RemoveContainer" containerID="c99e50eea3562218c0bb5c17d257f8fbe287413615ed42c5c59b5e6a7a33f71f" Oct 30 16:46:59.149343 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:46:59.149321 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n758f_openshift-console-operator(dacad254-b0c1-48fd-8377-1067d65ea230)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" podUID="dacad254-b0c1-48fd-8377-1067d65ea230" Oct 30 16:46:59.616226 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:59.616137 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:59.789225 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:46:59.789187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:00.107220 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:47:00.107183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:00.107412 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:47:00.107360 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:47:00.107554 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:47:00.107363 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:00.107554 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:47:00.107423 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:47:00.107554 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:47:00.107443 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:02.107420066 +0000 UTC m=+40.789623226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : secret "insights-runtime-extractor-tls" not found Oct 30 16:47:00.107554 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:47:00.107475 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:02.10746223 +0000 UTC m=+40.789665405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : configmap "kube-rbac-proxy" not found Oct 30 16:47:00.154384 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:47:00.154348 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-scdcl" event={"ID":"a132686f-9475-437d-826e-22fb93e86323","Type":"ContainerStarted","Data":"a6fa618ec5a7961862fe2d2935f3e1dcf1dab95c83f8a4b29da16c47f3bde923"} Oct 30 16:47:00.154856 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:47:00.154426 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:47:00.197733 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:47:00.197648 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-scdcl" podStartSLOduration=35.089162994 podStartE2EDuration="38.19763347s" totalImagesPullingTime="3.108470476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:00.197083024 +0000 UTC m=+38.879286205" watchObservedRunningTime="2026-10-30 16:47:00.19763347 +0000 UTC m=+38.879836650" Oct 30 16:47:00.460046 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:47:00.459952 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:01.677659 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:47:01.677627 2562 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:01.678043 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:47:01.677690 2562 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:01.678043 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:47:01.678038 2562 scope.go:122] "RemoveContainer" containerID="c99e50eea3562218c0bb5c17d257f8fbe287413615ed42c5c59b5e6a7a33f71f" Oct 30 16:47:01.678217 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:47:01.678200 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n758f_openshift-console-operator(dacad254-b0c1-48fd-8377-1067d65ea230)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" podUID="dacad254-b0c1-48fd-8377-1067d65ea230" Oct 30 16:47:02.125722 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:47:02.125694 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:02.125874 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:47:02.125754 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:02.125874 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:47:02.125823 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:47:02.125998 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:47:02.125877 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:06.125863366 +0000 UTC m=+44.808066541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : configmap "kube-rbac-proxy" not found Oct 30 16:47:02.125998 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:47:02.125928 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:47:02.126082 ip-10-0-135-153 kubenswrapper[2562]: E1030 16:47:02.126016 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:06.125998864 +0000 UTC m=+44.808202022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : secret "insights-runtime-extractor-tls" not found Oct 30 16:47:02.163393 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:47:02.163354 2562 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wgxv6" event={"ID":"ac8ed89d-9808-4bff-9d24-3e5fc34041ad","Type":"ContainerStarted","Data":"e3d7889201a106dd600f5930f472e3bf18a1390d04dc27d1f84aee3c120e9fd3"} Oct 30 16:47:02.215711 ip-10-0-135-153 kubenswrapper[2562]: I1030 16:47:02.215653 2562 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wgxv6" podStartSLOduration=34.147115861 podStartE2EDuration="40.215637913s" totalImagesPullingTime="6.068522052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:02.211007841 +0000 UTC m=+40.893211022" watchObservedRunningTime="2026-10-30 16:47:02.215637913 +0000 UTC m=+40.897841121" Oct 30 16:47:02.295573 ip-10-0-135-153 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 16:47:02.312315 ip-10-0-135-153 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 16:47:02.312484 ip-10-0-135-153 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 16:47:02.312535 ip-10-0-135-153 systemd[1]: kubelet.service: Consumed 3.054s CPU time. Oct 30 16:47:02.322520 ip-10-0-135-153 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:47:02.404240 ip-10-0-135-153 kubenswrapper[4828]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:47:02.404240 ip-10-0-135-153 kubenswrapper[4828]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:47:02.404240 ip-10-0-135-153 kubenswrapper[4828]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:47:02.404240 ip-10-0-135-153 kubenswrapper[4828]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:47:02.406636 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406535 4828 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:02.406636 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406635 4828 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:02.406636 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406639 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406643 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406646 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406649 4828 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406651 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406654 4828 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406657 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406660 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406663 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406665 4828 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406668 4828 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406671 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406674 4828 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406676 4828 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406679 4828 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406682 4828 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406684 4828 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406687 4828 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406689 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:02.406739 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406694 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406697 4828 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406700 4828 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406703 4828 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406706 4828 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406708 4828 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406719 4828 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406722 4828 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406725 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406727 4828 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406730 4828 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406732 4828 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406735 4828 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406738 4828 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406741 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406744 4828 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406747 4828 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406750 4828 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406753 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406756 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:02.407231 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406758 4828 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406761 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406764 4828 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406766 4828 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406770 4828 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406775 4828 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406779 4828 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406782 4828 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406785 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406788 4828 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406791 4828 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406793 4828 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406796 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406799 4828 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406802 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406805 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406807 4828 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406810 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406812 4828 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:02.407720 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406819 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406822 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406824 4828 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406827 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406830 4828 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406832 4828 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406835 4828 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406837 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406840 4828 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406842 4828 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406845 4828 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406847 4828 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406850 4828 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406852 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406855 4828 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406859 4828 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406861 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406864 4828 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406866 4828 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406869 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:02.408209 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406871 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406874 4828 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406876 4828 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406879 4828 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406882 4828 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406885 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406889 4828 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406892 4828 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406894 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406909 4828 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406912 4828 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406915 4828 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406917 4828 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406922 4828 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406925 4828 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406927 4828 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406931 4828 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406936 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406939 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:02.408691 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406941 4828 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406944 4828 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406946 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406949 4828 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406952 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406954 4828 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.406957 4828 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407749 4828 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407756 4828 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407759 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407763 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407765 4828 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407769 4828 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407771 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407774 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407777 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407779 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407782 4828 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407785 4828 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:02.409167 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407788 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407791 4828 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407794 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407796 4828 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407799 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407802 4828 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407805 4828 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407807 4828 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407811 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407813 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407816 4828 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407819 4828 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407821 4828 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407824 4828 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407827 4828 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407829 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407832 4828 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407834 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407837 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:02.409639 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407839 4828 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407842 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407845 4828 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407866 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407870 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407874 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407877 4828 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407880 4828 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407883 4828 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407885 4828 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407888 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407891 4828 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407895 4828 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407913 4828 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407916 4828 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407918 4828 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407921 4828 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407924 4828 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407926 4828 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407929 4828 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:02.410157 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407932 4828 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407934 4828 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407937 4828 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407940 4828 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407943 4828 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407946 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407948 4828 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407951 4828 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407954 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407957 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407960 4828 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407962 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407965 4828 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407967 4828 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407969 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407972 4828 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407976 4828 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407980 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407983 4828 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:02.410650 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407986 4828 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407989 4828 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407992 4828 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407994 4828 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.407997 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408000 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408002 4828 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408005 4828 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408007 4828 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408010 4828 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408013 4828 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408015 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408018 4828 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408020 4828 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408024 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408026 4828 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408029 4828 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408032 4828 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408035 4828 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408038 4828 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:02.411150 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408041 4828 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408043 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408046 4828 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408048 4828 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408050 4828 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408053 4828 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408056 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408058 4828 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408061 4828 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408064 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408066 4828 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408069 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408072 4828 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408074 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408077 4828 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408082 4828 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408596 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408601 4828 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408604 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:02.411667 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408607 4828 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408609 4828 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408612 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408615 4828 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408618 4828 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408621 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408624 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408627 4828 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408630 4828 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408633 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408636 4828 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408639 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408642 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408645 4828 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408647 4828 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408650 4828 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408653 4828 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408655 4828 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408658 4828 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:02.412140 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408661 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408663 4828 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408666 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408668 4828 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408671 4828 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408673 4828 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408677 4828 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408680 4828 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408683 4828 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408685 4828 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408688 4828 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408690 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408693 4828 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408696 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408699 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408701 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408704 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408707 4828 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408710 4828 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:02.412635 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408712 4828 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408715 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408717 4828 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408720 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408723 4828 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408725 4828 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408728 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408730 4828 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408733 4828 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408735 4828 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408738 4828 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408740 4828 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408743 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408745 4828 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408748 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408750 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408753 4828 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408756 4828 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408758 4828 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408761 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:02.413423 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408764 4828 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408766 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408769 4828 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408772 4828 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408774 4828 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408777 4828 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408779 4828 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408788 4828 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408790 4828 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408793 4828 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408795 4828 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408798 4828 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408801 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408804 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408806 4828 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408809 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408811 4828 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408814 4828 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408819 4828 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408823 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:02.414136 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408826 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408829 4828 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408832 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408836 4828 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408840 4828 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408842 4828 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408845 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408848 4828 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408851 4828 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408853 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408856 4828 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408859 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408862 4828 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408865 4828 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408867 4828 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408870 4828 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408873 4828 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408875 4828 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408878 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:02.414632 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408882 4828 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408884 4828 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408887 4828 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408889 4828 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408893 4828 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.408895 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409020 4828 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409039 4828 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409048 4828 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409053 4828 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409059 4828 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409063 4828 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409067 4828 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409071 4828 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409075 4828 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409078 4828 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409082 4828 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409085 4828 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409088 4828 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409092 4828 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409095 4828 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409099 4828 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409102 4828 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409105 4828 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:47:02.415126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409109 4828 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409113 4828 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409116 4828 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409119 4828 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409123 4828 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409127 4828 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409131 4828 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409134 4828 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409140 4828 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409143 4828 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409146 4828 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409149 4828 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409153 4828 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409156 4828 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409160 4828 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409163 4828 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409167 4828 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409170 4828 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409173 4828 flags.go:64] FLAG: --enable-server="true" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409176 4828 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409181 4828 flags.go:64] FLAG: --event-burst="100" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409185 4828 flags.go:64] FLAG: --event-qps="50" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409188 4828 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409191 4828 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:47:02.415711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409194 4828 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409198 4828 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409202 4828 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409205 4828 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409208 4828 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409211 4828 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409214 4828 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409217 4828 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409221 4828 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409224 4828 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409226 4828 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409229 4828 flags.go:64] FLAG: --feature-gates="" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409234 4828 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409237 4828 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409240 4828 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409244 4828 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409249 4828 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409253 4828 flags.go:64] FLAG: --help="false" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409255 4828 flags.go:64] FLAG: --hostname-override="ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409259 4828 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409262 4828 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409265 4828 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409268 4828 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:47:02.416323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409272 4828 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409275 4828 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409278 4828 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409283 4828 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409286 4828 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409289 4828 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409293 4828 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409296 4828 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409299 4828 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409303 4828 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409306 4828 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409309 4828 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409312 4828 flags.go:64] FLAG: --lock-file="" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409315 4828 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409318 4828 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409321 4828 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409327 4828 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409330 4828 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409334 4828 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409336 4828 flags.go:64] FLAG: --logging-format="text" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409339 4828 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409343 4828 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409347 4828 flags.go:64] FLAG: --manifest-url="" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409350 4828 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409355 4828 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:47:02.416877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409359 4828 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409364 4828 flags.go:64] FLAG: --max-pods="110" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409367 4828 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409370 4828 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409373 4828 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409376 4828 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409379 4828 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409382 4828 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409385 4828 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409392 4828 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409398 4828 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409401 4828 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409404 4828 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409407 4828 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409410 4828 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409413 4828 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409416 4828 flags.go:64] FLAG: --port="10250" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409420 4828 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409423 4828 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0404dcdd83f276fec" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409426 4828 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409429 4828 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409432 4828 flags.go:64] FLAG: --register-node="true" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409435 4828 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409439 4828 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409442 4828 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409444 4828 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:47:02.417499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409447 4828 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409451 4828 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409455 4828 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409458 4828 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409461 4828 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409464 4828 flags.go:64] FLAG: --runonce="false" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409468 4828 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409471 4828 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409474 4828 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409478 4828 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409481 4828 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409484 4828 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409487 4828 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409490 4828 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409493 4828 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409496 4828 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409500 4828 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409503 4828 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409507 4828 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409510 4828 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409513 4828 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409519 4828 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409522 4828 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409525 4828 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409528 4828 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:47:02.418157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409531 4828 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409534 4828 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409537 4828 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409540 4828 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409543 4828 flags.go:64] FLAG: --v="2" Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409553 4828 flags.go:64] FLAG: --version="false" Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409557 4828 flags.go:64] FLAG: --vmodule="" Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409561 4828 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.409565 4828 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409714 4828 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409718 4828 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409721 4828 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409724 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409728 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409731 4828 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409734 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409736 4828 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409739 4828 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409742 4828 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409744 4828 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409747 4828 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:02.418788 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409749 4828 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409752 4828 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409756 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409759 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409762 4828 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409764 4828 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409767 4828 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409770 4828 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409773 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409776 4828 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409779 4828 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409781 4828 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409784 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409786 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409789 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409792 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409794 4828 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409797 4828 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409799 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:02.419379 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409802 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409806 4828 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409810 4828 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409813 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409816 4828 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409820 4828 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409822 4828 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409825 4828 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409828 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409830 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409833 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409836 4828 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409838 4828 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409841 4828 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409844 4828 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409847 4828 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409850 4828 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409853 4828 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409855 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409858 4828 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:02.419835 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409861 4828 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409863 4828 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409866 4828 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409869 4828 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409871 4828 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409874 4828 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409877 4828 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409879 4828 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409882 4828 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409884 4828 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409887 4828 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409889 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409892 4828 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409894 4828 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409909 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409911 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409914 4828 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409918 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409921 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:02.420321 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409924 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409926 4828 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409929 4828 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409932 4828 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409934 4828 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409937 4828 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409940 4828 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409942 4828 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409946 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409949 4828 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409951 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409954 4828 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409956 4828 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409959 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409961 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409964 4828 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409967 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409970 4828 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409973 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:02.420798 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409975 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409978 4828 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409980 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409983 4828 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409986 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409988 4828 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409991 4828 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409993 4828 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.409997 4828 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410000 4828 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410004 4828 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410008 4828 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410011 4828 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410013 4828 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410016 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410019 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:02.421342 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410021 4828 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.410048 4828 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410085 4828 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410089 4828 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410092 4828 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410096 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410099 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410101 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410104 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410106 4828 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410109 4828 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410111 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410114 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410118 4828 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410123 4828 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410126 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:02.421974 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410129 4828 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410131 4828 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410134 4828 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410137 4828 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410139 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410142 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410145 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410147 4828 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410150 4828 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410152 4828 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410155 4828 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410158 4828 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410162 4828 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410165 4828 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410168 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410170 4828 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410173 4828 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410175 4828 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410178 4828 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410181 4828 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:02.422454 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410184 4828 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410187 4828 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410190 4828 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410192 4828 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410195 4828 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410198 4828 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410200 4828 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410203 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410206 4828 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410209 4828 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410211 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410214 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410217 4828 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410219 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410222 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410225 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410227 4828 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410230 4828 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410233 4828 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410236 4828 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:02.423076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410239 4828 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410241 4828 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410244 4828 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410246 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410250 4828 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410253 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410256 4828 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410258 4828 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410261 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410263 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410266 4828 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410269 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410271 4828 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410275 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410277 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410280 4828 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410283 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410285 4828 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410288 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:02.423627 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410290 4828 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410293 4828 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410296 4828 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410299 4828 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410303 4828 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410307 4828 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410310 4828 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410313 4828 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410316 4828 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410318 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410321 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410323 4828 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410326 4828 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410329 4828 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410331 4828 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410334 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410336 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410340 4828 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410343 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:02.424325 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410345 4828 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410348 4828 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410351 4828 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410353 4828 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410357 4828 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410359 4828 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410362 4828 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410366 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410369 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410373 4828 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410375 4828 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410378 4828 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410381 4828 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.410383 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.415723 4828 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.415744 4828 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.415766 4828 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.415771 4828 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415839 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:02.424877 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415844 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415847 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415850 4828 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415853 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415856 4828 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415859 4828 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415862 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415864 4828 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415868 4828 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415870 4828 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415873 4828 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415876 4828 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415878 4828 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415886 4828 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415889 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415892 4828 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415895 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415912 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415916 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415918 4828 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:02.425361 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415921 4828 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415924 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415927 4828 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415930 4828 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415933 4828 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415936 4828 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415939 4828 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415941 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415944 4828 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415947 4828 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415949 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415953 4828 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415955 4828 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415958 4828 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415961 4828 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415964 4828 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415967 4828 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415969 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415972 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:02.425842 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415974 4828 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415977 4828 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415980 4828 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415983 4828 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415986 4828 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415988 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415992 4828 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415994 4828 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.415997 4828 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416000 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416002 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416005 4828 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416007 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416010 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416012 4828 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416015 4828 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416018 4828 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416021 4828 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416024 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:02.426320 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416026 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416029 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416032 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416034 4828 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416037 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416040 4828 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416042 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416045 4828 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416047 4828 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416052 4828 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416056 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416059 4828 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416062 4828 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416065 4828 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416068 4828 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416071 4828 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416074 4828 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416077 4828 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416079 4828 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416082 4828 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:02.426817 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416085 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416088 4828 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416091 4828 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416094 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416097 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416100 4828 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416104 4828 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416107 4828 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416110 4828 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416113 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416116 4828 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416119 4828 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416122 4828 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416125 4828 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416127 4828 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416130 4828 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416133 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416135 4828 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416138 4828 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416140 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:02.427345 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416143 4828 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416145 4828 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416148 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416151 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416153 4828 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416156 4828 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416158 4828 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.416188 4828 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416345 4828 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416350 4828 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416354 4828 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416357 4828 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416360 4828 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416367 4828 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416371 4828 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416375 4828 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:02.427853 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416378 4828 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416381 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416384 4828 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416386 4828 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416389 4828 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416392 4828 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416395 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416397 4828 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416400 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416402 4828 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416405 4828 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416407 4828 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416410 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416413 4828 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416415 4828 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416418 4828 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416420 4828 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416423 4828 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416425 4828 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416428 4828 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:02.428260 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416431 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416433 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416436 4828 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416439 4828 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416442 4828 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416445 4828 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416447 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416450 4828 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416452 4828 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416455 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416459 4828 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416461 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416464 4828 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416467 4828 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416469 4828 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416472 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416475 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416477 4828 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416480 4828 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416483 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:02.429076 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416486 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416488 4828 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416491 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416493 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416496 4828 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416498 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416501 4828 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416503 4828 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416506 4828 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416509 4828 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416511 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416514 4828 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416517 4828 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416519 4828 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416522 4828 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416524 4828 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416527 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416529 4828 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416532 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416534 4828 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:02.429752 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416537 4828 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416540 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416543 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416546 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416549 4828 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416551 4828 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416554 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416556 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416559 4828 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416562 4828 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416564 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416567 4828 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416570 4828 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416574 4828 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416578 4828 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416581 4828 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416584 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416586 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416589 4828 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:02.430384 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416592 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416594 4828 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416597 4828 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416599 4828 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416602 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416605 4828 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416607 4828 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416610 4828 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416613 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416615 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416618 4828 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416621 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416623 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416626 4828 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416629 4828 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416632 4828 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416634 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416637 4828 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:02.430915 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416640 4828 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.416663 4828 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416696 4828 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416700 4828 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416703 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416705 4828 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416708 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416711 4828 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416714 4828 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416716 4828 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416719 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416722 4828 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416724 4828 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416727 4828 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416729 4828 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416732 4828 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:47:02.431530 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416734 4828 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416737 4828 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416740 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416742 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416745 4828 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416747 4828 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416750 4828 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416753 4828 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416755 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416758 4828 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416761 4828 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416763 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416766 4828 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416769 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416771 4828 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416773 4828 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416776 4828 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416779 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416782 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416784 4828 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:47:02.432077 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416787 4828 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416790 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416792 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416795 4828 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416797 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416800 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416802 4828 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416805 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416807 4828 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416810 4828 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416812 4828 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416815 4828 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416818 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416821 4828 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416823 4828 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416826 4828 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416828 4828 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416831 4828 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416833 4828 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416836 4828 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:47:02.432669 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416839 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416841 4828 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416844 4828 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416846 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416849 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416851 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416854 4828 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416856 4828 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416859 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416862 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416865 4828 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416867 4828 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416870 4828 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416872 4828 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416875 4828 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416879 4828 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416884 4828 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416888 4828 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416891 4828 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:47:02.433404 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416894 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416909 4828 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416913 4828 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416916 4828 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416918 4828 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416923 4828 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416926 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416929 4828 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416932 4828 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416935 4828 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416938 4828 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416940 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416943 4828 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416945 4828 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416948 4828 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416951 4828 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416953 4828 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416956 4828 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416959 4828 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:47:02.434096 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416961 4828 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416964 4828 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416966 4828 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416969 4828 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416972 4828 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416975 4828 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416977 4828 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416980 4828 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416985 4828 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416988 4828 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416991 4828 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416993 4828 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416996 4828 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:02.416999 4828 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.417063 4828 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.418085 4828 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.418158 4828 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.418339 4828 server.go:1014] "Starting client certificate rotation" Oct 30 16:47:02.434615 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.418473 4828 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:47:02.435202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.418525 4828 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:41:21 +0000 UTC" deadline="2028-06-19 20:20:12.27999389 +0000 UTC" Oct 30 16:47:02.435202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.418553 4828 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14355h33m9.861442178s" Oct 30 16:47:02.435202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.418813 4828 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:47:02.435202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.419960 4828 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:47:02.435202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.421543 4828 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:47:02.435202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.423401 4828 log.go:25] "Validated CRI v1 image API" Oct 30 16:47:02.435202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.424132 4828 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:47:02.435202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.426349 4828 fs.go:135] Filesystem UUIDs: map[7144d48a-3974-4cc6-9382-77cb4226ad18:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b137222b-6488-4ff5-8450-70be74d5ba21:/dev/nvme0n1p4] Oct 30 16:47:02.435422 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.426370 4828 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/035c8825d19d48aecd545e8171275583b3451316fc2fadf02d18814fa6f0a578/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/035c8825d19d48aecd545e8171275583b3451316fc2fadf02d18814fa6f0a578/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03ebb684cd0cb48ec84a17364457a71e57eba2689747985e90df2b2ed4da1303/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03ebb684cd0cb48ec84a17364457a71e57eba2689747985e90df2b2ed4da1303/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06bbf6f61b78f3ae3784d719972af3817a864582ffed7a1bd6f7e5b92d011189/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06bbf6f61b78f3ae3784d719972af3817a864582ffed7a1bd6f7e5b92d011189/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b166ba605a74b057cd8ab0032f9e35754ce05745876024b528a0b101fba0e4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b166ba605a74b057cd8ab0032f9e35754ce05745876024b528a0b101fba0e4c/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0cb0b06adaba74a0251d98b36fe0ad9508165fa9cc4a372a21cfcef74e6f9967/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0cb0b06adaba74a0251d98b36fe0ad9508165fa9cc4a372a21cfcef74e6f9967/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1be7b177c8467d07515635775555327fa59b33db1f3cf2100c5f2e6ef349493d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1be7b177c8467d07515635775555327fa59b33db1f3cf2100c5f2e6ef349493d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1da9fa642e9eaa3fbf7233d736ecacb10e621a2130b022ea6b1b02ab1aa1f2f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1da9fa642e9eaa3fbf7233d736ecacb10e621a2130b022ea6b1b02ab1aa1f2f0/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e59114d4ba6e950fd87cd095372196c103e71fa14acf4b788c9d916fe99b0a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e59114d4ba6e950fd87cd095372196c103e71fa14acf4b788c9d916fe99b0a4/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2baff9f0fa3824f1aab8765d996070f22b46a05fe7497b4026d9d8798008ac70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2baff9f0fa3824f1aab8765d996070f22b46a05fe7497b4026d9d8798008ac70/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d9a4c93cd8726956f1a8f7901706b5a824a9ce074b05b8adb54875dd8ecd2bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d9a4c93cd8726956f1a8f7901706b5a824a9ce074b05b8adb54875dd8ecd2bd/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/612ff75fe18ddb930ae5640e590053e6d754dd2a49f4c9f0a7259c66b8f97ff2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/612ff75fe18ddb930ae5640e590053e6d754dd2a49f4c9f0a7259c66b8f97ff2/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/774df0d8e22e6bacf27357b34b44bd971a3a2680b7bcb2bfd028b632d778ea35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/774df0d8e22e6bacf27357b34b44bd971a3a2680b7bcb2bfd028b632d778ea35/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77affc7483735aab7d4b37a8768bcd98379bf5cc9f4d73e893cf3e1fd40f69bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77affc7483735aab7d4b37a8768bcd98379bf5cc9f4d73e893cf3e1fd40f69bb/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a196d593fd53d6c5c9b76e4ba832465346894a7262a169b1b18c344b594b65c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a196d593fd53d6c5c9b76e4ba832465346894a7262a169b1b18c344b594b65c9/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3cc7e5082a0f483f87a3aff41083d0f9e6f8989b11dcb35e38ef070a1f7bb0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3cc7e5082a0f483f87a3aff41083d0f9e6f8989b11dcb35e38ef070a1f7bb0a/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7b568d421f6a612f9dd683fb4c69f0afba11581168228b603e39f73a03a9c6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7b568d421f6a612f9dd683fb4c69f0afba11581168228b603e39f73a03a9c6a/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3c7cd045ffa4e727bde3156153e89aa8677f1e94e56a90ceec2ba11980727bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3c7cd045ffa4e727bde3156153e89aa8677f1e94e56a90ceec2ba11980727bf/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d210d1881661386240975f06970c0bb1710c5358b3d15c73a1eb91b9f52dcbcf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d210d1881661386240975f06970c0bb1710c5358b3d15c73a1eb91b9f52dcbcf/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7120dbf5c69333bd3e5babceeda19cf41131e73908d132cd7c31a3b7b882137/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7120dbf5c69333bd3e5babceeda19cf41131e73908d132cd7c31a3b7b882137/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbe6588c13763b48fd28430b4f4c5429731f4c47e52fee7e9a79177f59fcfbd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbe6588c13763b48fd28430b4f4c5429731f4c47e52fee7e9a79177f59fcfbd0/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecbf1d255616640b88798eb88bb8b0b048bfc7bc5cca973fa973d826ee4f92e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecbf1d255616640b88798eb88bb8b0b048bfc7bc5cca973fa973d826ee4f92e8/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f18612a92e2dc29542be8b2489f9ff7c492ee84b563208eb8fbfba4460c0aecc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f18612a92e2dc29542be8b2489f9ff7c492ee84b563208eb8fbfba4460c0aecc/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e/volumes/kubernetes.io~projected/kube-api-access-z8f6w:{mountpoint:/var/lib/kubelet/pods/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e/volumes/kubernetes.io~projected/kube-api-access-z8f6w major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/159cde9b-22a5-4930-9bf6-2535ab4af3d4/volumes/kubernetes.io~projected/kube-api-access-p4x9q:{mountpoint:/var/lib/kubelet/pods/159cde9b-22a5-4930-9bf6-2535ab4af3d4/volumes/kubernetes.io~projected/kube-api-access-p4x9q major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/252c7435-f4c1-4dac-a468-e3287c6a7e9e/volumes/kubernetes.io~projected/kube-api-access-774sr:{mountpoint:/var/lib/kubelet/pods/252c7435-f4c1-4dac-a468-e3287c6a7e9e/volumes/kubernetes.io~projected/kube-api-access-774sr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30001919-517f-4031-9ef4-bc0fbb43d122/volumes/kubernetes.io~projected/kube-api-access-4tnhb:{mountpoint:/var/lib/kubelet/pods/30001919-517f-4031-9ef4-bc0fbb43d122/volumes/kubernetes.io~projected/kube-api-access-4tnhb major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3095c033-c4ff-42cd-9530-1be17a144a31/volumes/kubernetes.io~projected/kube-api-access-fgpkr:{mountpoint:/var/lib/kubelet/pods/3095c033-c4ff-42cd-9530-1be17a144a31/volumes/kubernetes.io~projected/kube-api-access-fgpkr major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35a14012-73eb-4f3b-b762-f56c6a963e44/volumes/kubernetes.io~projected/kube-api-access-sgkdb:{mountpoint:/var/lib/kubelet/pods/35a14012-73eb-4f3b-b762-f56c6a963e44/volumes/kubernetes.io~projected/kube-api-access-sgkdb major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35a14012-73eb-4f3b-b762-f56c6a963e44/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/35a14012-73eb-4f3b-b762-f56c6a963e44/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3794a50e-6c2f-40bc-9be7-eda1388f3178/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3794a50e-6c2f-40bc-9be7-eda1388f3178/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3794a50e-6c2f-40bc-9be7-eda1388f3178/volumes/kubernetes.io~projected/kube-api-access-7xjxv:{mountpoint:/var/lib/kubelet/pods/3794a50e-6c2f-40bc-9be7-eda1388f3178/volumes/kubernetes.io~projected/kube-api-access-7xjxv major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3794a50e-6c2f-40bc-9be7-eda1388f3178/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3794a50e-6c2f-40bc-9be7-eda1388f3178/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3794a50e-6c2f-40bc-9be7-eda1388f3178/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3794a50e-6c2f-40bc-9be7-eda1388f3178/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ca4c236-e28d-4331-9336-63a71d706ef0/volumes/kubernetes.io~projected/kube-api-access-svsvb:{mountpoint:/var/lib/kubelet/pods/3ca4c236-e28d-4331-9336-63a71d706ef0/volumes/kubernetes.io~projected/kube-api-access-svsvb major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cdefc81-0135-4053-bc7b-78958686e8ea/volumes/kubernetes.io~projected/kube-api-access-64l24:{mountpoint:/var/lib/kubelet/pods/3cdefc81-0135-4053-bc7b-78958686e8ea/volumes/kubernetes.io~projected/kube-api-access-64l24 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68321123-f8fc-4c6a-bd22-1f614a8a3b55/volumes/kubernetes.io~projected/kube-api-access-qprsq:{mountpoint:/var/lib/kubelet/pods/68321123-f8fc-4c6a-bd22-1f614a8a3b55/volumes/kubernetes.io~projected/kube-api-access-qprsq major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68321123-f8fc-4c6a-bd22-1f614a8a3b55/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/68321123-f8fc-4c6a-bd22-1f614a8a3b55/volumes/kubernetes.io~secret/cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/852f74bb-c91e-4075-8335-aeba3e723709/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/852f74bb-c91e-4075-8335-aeba3e723709/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/852f74bb-c91e-4075-8335-aeba3e723709/volumes/kubernetes.io~projected/kube-api-access-vp752:{mountpoint:/var/lib/kubelet/pods/852f74bb-c91e-4075-8335-aeba3e723709/volumes/kubernetes.io~projected/kube-api-access-vp752 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/852f74bb-c91e-4075-8335-aeba3e723709/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/852f74bb-c91e-4075-8335-aeba3e723709/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/883b14e2-b8bc-4650-a638-c33673683a45/volumes/kubernetes.io~projected/kube-api-access-fwc77:{mountpoint:/var/lib/kubelet/pods/883b14e2-b8bc-4650-a638-c33673683a45/volumes/kubernetes.io~projected/kube-api-access-fwc77 major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c30c30a-8c1f-4552-b2d1-2692bc423d45/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8c30c30a-8c1f-4552-b2d1-2692bc423d45/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c30c30a-8c1f-4552-b2d1-2692bc423d45/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8c30c30a-8c1f-4552-b2d1-2692bc423d45/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c30c30a-8c1f-4552-b2d1-2692bc423d45/volumes/kubernetes.io~projected/kube-api-access-9nttq:{mountpoint:/var/lib/kubelet/pods/8c30c30a-8c1f-4552-b2d1-2692bc423d45/volumes/kubernetes.io~projected/kube-api-access-9nttq major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a132686f-9475-437d-826e-22fb93e86323/volumes/kubernetes.io~projected/kube-api-access-cmzz5:{mountpoint:/var/lib/kubelet/pods/a132686f-9475-437d-826e-22fb93e86323/volumes/kubernetes.io~projected/kube-api-access-cmzz5 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a41c35af-f159-4802-9b53-fbb8cbee2104/volumes/kubernetes.io~projected/kube-api-access-rvsrx:{mountpoint:/var/lib/kubelet/pods/a41c35af-f159-4802-9b53-fbb8cbee2104/volumes/kubernetes.io~projected/kube-api-access-rvsrx major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40/volumes/kubernetes.io~projected/kube-api-access-pjrlc:{mountpoint:/var/lib/kubelet/pods/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40/volumes/kubernetes.io~projected/kube-api-access-pjrlc major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40/volumes/kubernetes.io~secret/stats-auth major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac8ed89d-9808-4bff-9d24-3e5fc34041ad/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ac8ed89d-9808-4bff-9d24-3e5fc34041ad/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b174c80c-af6a-4f17-8442-1a029b286011/volumes/kubernetes.io~projected/kube-api-access-6w2qd:{mountpoint:/var/lib/kubelet/pods/b174c80c-af6a-4f17-8442-1a029b286011/volumes/kubernetes.io~projected/kube-api-access-6w2qd major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b34094c9-7349-4284-8b86-2253f9e4d87f/volumes/kubernetes.io~projected/kube-api-access-zmgvx:{mountpoint:/var/lib/kubelet/pods/b34094c9-7349-4284-8b86-2253f9e4d87f/volumes/kubernetes.io~projected/kube-api-access-zmgvx major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b516f174-836e-4711-8999-80ce5a592e28/volumes/kubernetes.io~projected/kube-api-access-q6zs9:{mountpoint:/var/lib/kubelet/pods/b516f174-836e-4711-8999-80ce5a592e28/volumes/kubernetes.io~projected/kube-api-access-q6zs9 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b516f174-836e-4711-8999-80ce5a592e28/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b516f174-836e-4711-8999-80ce5a592e28/volumes/kubernetes.io~secret/serving-cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b54c34b7-e8c3-4fe9-9b66-36336dfae405/volumes/kubernetes.io~projected/kube-api-access-4l2kn:{mountpoint:/var/lib/kubelet/pods/b54c34b7-e8c3-4fe9-9b66-36336dfae405/volumes/kubernetes.io~projected/kube-api-access-4l2kn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d38312b4-bd6e-4a5f-8a33-d93fb827debc/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d38312b4-bd6e-4a5f-8a33-d93fb827debc/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d38312b4-bd6e-4a5f-8a33-d93fb827debc/volumes/kubernetes.io~projected/kube-api-access-q52w4:{mountpoint:/var/lib/kubelet/pods/d38312b4-bd6e-4a5f-8a33-d93fb827debc/volumes/kubernetes.io~projected/kube-api-access-q52w4 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d38312b4-bd6e-4a5f-8a33-d93fb827debc/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d38312b4-bd6e-4a5f-8a33-d93fb827debc/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d38312b4-bd6e-4a5f-8a33-d93fb827debc/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d38312b4-bd6e-4a5f-8a33-d93fb827debc/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7a04182-8031-4fa8-8bed-4f8b60b6cc97/volumes/kubernetes.io~projected/kube-api-access-4srgz:{mountpoint:/var/lib/kubelet/pods/d7a04182-8031-4fa8-8bed-4f8b60b6cc97/volumes/kubernetes.io~projected/kube-api-access-4srgz major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dacad254-b0c1-48fd-8377-1067d65ea230/volumes/kubernetes.io~projected/kube-api-access-8cmkh:{mountpoint:/var/lib/kubelet/pods/dacad254-b0c1-48fd-8377-1067d65ea230/volumes/kubernetes.io~projected/kube-api-access-8cmkh major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dacad254-b0c1-48fd-8377-1067d65ea230/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/dacad254-b0c1-48fd-8377-1067d65ea230/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbb5e2c6-44f4-4152-a7ac-98ae4c045190/volumes/kubernetes.io~projected/kube-api-access-gm5w2:{mountpoint:/var/lib/kubelet/pods/dbb5e2c6-44f4-4152-a7ac-98ae4c045190/volumes/kubernetes.io~projected/kube-api-access-gm5w2 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f/volumes/kubernetes.io~projected/kube-api-access-rvwhw:{mountpoint:/var/lib/kubelet/pods/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f/volumes/kubernetes.io~projected/kube-api-access-rvwhw major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f13fd623-29b9-4d8b-b76d-668ca123010f/volumes/kubernetes.io~projected/kube-api-access-sqff6:{mountpoint:/var/lib/kubelet/pods/f13fd623-29b9-4d8b-b76d-668ca123010f/volumes/kubernetes.io~projected/kube-api-access-sqff6 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8aeb454-5336-4fc1-8194-54d54a6cdb6c/volumes/kubernetes.io~projected/kube-api-access-72lmm:{mountpoint:/var/lib/kubelet/pods/f8aeb454-5336-4fc1-8194-54d54a6cdb6c/volumes/kubernetes.io~projected/kube-api-access-72lmm major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/27cda951c5d6729843702e695fcf09bf6190f98829ed6eeb2255410ee65c486e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f4c19df51d0e6ccd4fa5c9b5e368a7f950fcdc325ac357aa070946c6744abad8/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/fa8ef96bf9ccec68fd584426e28283870af29f8c31cbe0953b6e9fc4b4e6e262/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/3bfcc72354dac0be5094ea1442f7b0e0646d0a5369e8d0fdb2336cef9934e3e9/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/7281e1efba93db651739b983740a23256a7cc2102636af4ee77187693770a295/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/81ce863de507c2dd101a08937e03f6ce1aca8d97e47976b695354dc24ebe5abf/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7ab8d0ca2fba13eb911d818beebdae639a86c211d79aba6c1e7445bb6bf1df7e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/524f27053837891fd44122de74d643c1b8059076c6d29c7d5eca299a20ddac3f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/c49db574cdced7b39386830e876c6fa13ef5bee599f4443c871d622e926f9eed/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/19381254e8ac9202e3327cd609106afb1a1acde5e22efb9134662fe7004d0c1c/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/fbab4f93a4a1a7a691f4da9173bf7d1c3f92c7f0f7c6af1b5ec97db4d2e5844d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/daa6aae7834f28e9e2c2a29fc08c1203e4f0bb372cd9f41c59899dbe8dcb1031/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/647a9a1e4152ea60ed85c65a89efaab518b33d58b747e591d277e34c26e19cce/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c5bd8e5f1a302af8975f2b63df009db7d1d7ba4c8bc3ed39ef4a705c0c9d27bf/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/8b774e9cb336504d6e36afeab64341e8d883cabb604b055ea31b763f389d3a98/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/28c6366d93094962c13ffdc2c83de563bddad63ab9723686c916e85e864db77c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6d5a028d0e881b6d54cfb3603bfc0ee32c50479b246d8a62a627f19798a018e6/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b3ccbad30504ee3647a837cbe6619d423c8492303af7c7fc52ce23bfa3f1a558/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c6a7803a1a52ca04965e49d8bf2394d555a67936eb9dc74748d133c51719a9cd/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e08895116313c42839eff83b269752c66f1049db86532f75c0055c9a81011591/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/fb98942a41208814c5b123d48bd6b159e429d960179fa83fcf6ab5c836b66dbc/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/6698653d76c205243950b9fb5a64ff98469d5b6d61e2f7fc602641bb0f567434/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/2ca5c0f0d60987547ccf178802f3ef488e20ce73748399d57150500172423dcd/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/2d358261b775d9bf7635b2bbe17d77f431b18248dbf9bc2a0b9db16f68322a07/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/f38ac404410376388e7280c784200c0293320a08d163994383ca11b4f37b922f/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/8858df0578062aad353a1b49e7726e5b08319b44b23ff10f4a3eded973598f4f/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/a100669cb53eb655ba3347f376ec2d666962b831b4b79211e6298c603d6d1ec4/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/f9d46a0efb613a4da3695b2ed228b702c8a5bc1dcc0e6c14e1fb2e3042ddf5f7/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/04b54946b0c93bb838380b77bd74730412ae84a1684e443cee5f6614b90871d5/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/b8c235284ddd60c7941eeead4366ebb694c51a74d0340a8fba8cfb577788780c/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/4e892ebe67546620a5fcca4f19b1dbe7da0615672849a3fb85fbc14ea3b5969f/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/bec489a43351d77462152cdb344ee5fc6531e9fd8152f75faf553be25388d5d9/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/eecaa36bcda40eb85346120152cce6ab2a6e50e8db037c7d2652d0681be48569/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/648d8ac74ba64c1bdc52dae5cce24093ea8a3ac2cb3d1b818c2218b3908371ec/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/693cfbdcb59e01eeec8826e67956adcc3ceac62fe48cd12422b04ecdb73ac686/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/df2955f53c05460a0d3901625222925f1c90214df261adb2e633a68f9e5bb64b/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/aaeca4aebd256dd68def502a8c191d2807afda089b43189909b6831d02301e87/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/527f23ef2f35079495b6d86172875f23906ee6045a4735ec3629934090ade906/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/96d499d8583955aedae7f94e2d94b1a96e3db722d1713811b112dffb40cb2472/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/f6d6fc5b23acb942b559868c5bb7b96ff9cf396d9829a22ad970155acbf43886/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/1e5de318a548fc031bce72bead69383669f1930833959bf99bb4f4a23c189613/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/d206c2f57762811138eaa6ff904615bfd24387cd5023ff46f149087d8521a01e/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/2565cf1aa8ae8f5d96c2ab014cc92fb8e9d00fbf2d1f19c933ea5dd0891b89b5/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/c5840d2a4efb42d37dcb5796e1a6c6846c827736b47c35ca18515186e084d9d1/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/d1c5b809737be38705c70537354b017effe6db712df08b834a4f6f3d5bed48b4/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/9ecd76f6f3f77474982cab7b0fad324e36386c3e23bc2ac95142bb6495412333/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d5eee9ee7c4e0f4d22f2af2774495947691c071d25300293ed6c4e6dbf7ab360/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/4b2114d15951c3234fac60b38849b5ea82783bfd6b8afdbd5d80c1e30e5ef7c4/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/97a134367ba8dbb58ae99b1082b78b1bc9efda61821c64f654b56f41142ce802/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/6baf8377d9b81090428388ba805a265fe8e3558095bb57fe036b5031623ac7f0/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/2b5b979cb978c885ebb39c975da9434d855682ae980a38f943228019c7a3b1f3/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/cd6481818f319696c6e4884a9b529b801578562578969c978cd484584e498d82/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d93b97f7623df413146d06b5541bc9ceb2cd6bb14bc796e68ce9473c1d81009c/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/aab85ee516182ddf9972087f6aabc7d4c6177c474401010213cce4983f52f187/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 16:47:02.435633 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.434871 4828 manager.go:217] Machine: {Timestamp:2026-10-30 16:47:02.434399847 +0000 UTC m=+0.079910019 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100307 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20c9094790d091b2a7c2bb0d4f7aac SystemUUID:ec20c909-4790-d091-b2a7-c2bb0d4f7aac BootID:8b035d1e-6263-40d4-b37d-0da56a929adb Filesystems:[{Device:/run/containers/storage/overlay-containers/035c8825d19d48aecd545e8171275583b3451316fc2fadf02d18814fa6f0a578/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68321123-f8fc-4c6a-bd22-1f614a8a3b55/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/159cde9b-22a5-4930-9bf6-2535ab4af3d4/volumes/kubernetes.io~projected/kube-api-access-p4x9q DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/35a14012-73eb-4f3b-b762-f56c6a963e44/volumes/kubernetes.io~projected/kube-api-access-sgkdb DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/883b14e2-b8bc-4650-a638-c33673683a45/volumes/kubernetes.io~projected/kube-api-access-fwc77 DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06bbf6f61b78f3ae3784d719972af3817a864582ffed7a1bd6f7e5b92d011189/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3794a50e-6c2f-40bc-9be7-eda1388f3178/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7b568d421f6a612f9dd683fb4c69f0afba11581168228b603e39f73a03a9c6a/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e/volumes/kubernetes.io~projected/kube-api-access-z8f6w DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3794a50e-6c2f-40bc-9be7-eda1388f3178/volumes/kubernetes.io~projected/kube-api-access-7xjxv DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b34094c9-7349-4284-8b86-2253f9e4d87f/volumes/kubernetes.io~projected/kube-api-access-zmgvx DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ca4c236-e28d-4331-9336-63a71d706ef0/volumes/kubernetes.io~projected/kube-api-access-svsvb DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d38312b4-bd6e-4a5f-8a33-d93fb827debc/volumes/kubernetes.io~projected/kube-api-access-q52w4 DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/68321123-f8fc-4c6a-bd22-1f614a8a3b55/volumes/kubernetes.io~projected/kube-api-access-qprsq DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dbb5e2c6-44f4-4152-a7ac-98ae4c045190/volumes/kubernetes.io~projected/kube-api-access-gm5w2 DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/612ff75fe18ddb930ae5640e590053e6d754dd2a49f4c9f0a7259c66b8f97ff2/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3794a50e-6c2f-40bc-9be7-eda1388f3178/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dacad254-b0c1-48fd-8377-1067d65ea230/volumes/kubernetes.io~projected/kube-api-access-8cmkh DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f/volumes/kubernetes.io~projected/kube-api-access-rvwhw DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03ebb684cd0cb48ec84a17364457a71e57eba2689747985e90df2b2ed4da1303/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7120dbf5c69333bd3e5babceeda19cf41131e73908d132cd7c31a3b7b882137/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d38312b4-bd6e-4a5f-8a33-d93fb827debc/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f13fd623-29b9-4d8b-b76d-668ca123010f/volumes/kubernetes.io~projected/kube-api-access-sqff6 DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/852f74bb-c91e-4075-8335-aeba3e723709/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a132686f-9475-437d-826e-22fb93e86323/volumes/kubernetes.io~projected/kube-api-access-cmzz5 DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3cc7e5082a0f483f87a3aff41083d0f9e6f8989b11dcb35e38ef070a1f7bb0a/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/252c7435-f4c1-4dac-a468-e3287c6a7e9e/volumes/kubernetes.io~projected/kube-api-access-774sr DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e59114d4ba6e950fd87cd095372196c103e71fa14acf4b788c9d916fe99b0a4/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/852f74bb-c91e-4075-8335-aeba3e723709/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a196d593fd53d6c5c9b76e4ba832465346894a7262a169b1b18c344b594b65c9/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f8aeb454-5336-4fc1-8194-54d54a6cdb6c/volumes/kubernetes.io~projected/kube-api-access-72lmm DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d38312b4-bd6e-4a5f-8a33-d93fb827debc/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3095c033-c4ff-42cd-9530-1be17a144a31/volumes/kubernetes.io~projected/kube-api-access-fgpkr DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8c30c30a-8c1f-4552-b2d1-2692bc423d45/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecbf1d255616640b88798eb88bb8b0b048bfc7bc5cca973fa973d826ee4f92e8/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40/volumes/kubernetes.io~projected/kube-api-access-pjrlc DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/35a14012-73eb-4f3b-b762-f56c6a963e44/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1da9fa642e9eaa3fbf7233d736ecacb10e621a2130b022ea6b1b02ab1aa1f2f0/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f18612a92e2dc29542be8b2489f9ff7c492ee84b563208eb8fbfba4460c0aecc/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7a04182-8031-4fa8-8bed-4f8b60b6cc97/volumes/kubernetes.io~projected/kube-api-access-4srgz DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3cdefc81-0135-4053-bc7b-78958686e8ea/volumes/kubernetes.io~projected/kube-api-access-64l24 DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac8ed89d-9808-4bff-9d24-3e5fc34041ad/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a41c35af-f159-4802-9b53-fbb8cbee2104/volumes/kubernetes.io~projected/kube-api-access-rvsrx DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b54c34b7-e8c3-4fe9-9b66-36336dfae405/volumes/kubernetes.io~projected/kube-api-access-4l2kn DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0cb0b06adaba74a0251d98b36fe0ad9508165fa9cc4a372a21cfcef74e6f9967/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77affc7483735aab7d4b37a8768bcd98379bf5cc9f4d73e893cf3e1fd40f69bb/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8c30c30a-8c1f-4552-b2d1-2692bc423d45/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8c30c30a-8c1f-4552-b2d1-2692bc423d45/volumes/kubernetes.io~projected/kube-api-access-9nttq DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d9a4c93cd8726956f1a8f7901706b5a824a9ce074b05b8adb54875dd8ecd2bd/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/852f74bb-c91e-4075-8335-aeba3e723709/volumes/kubernetes.io~projected/kube-api-access-vp752 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2baff9f0fa3824f1aab8765d996070f22b46a05fe7497b4026d9d8798008ac70/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1be7b177c8467d07515635775555327fa59b33db1f3cf2100c5f2e6ef349493d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d38312b4-bd6e-4a5f-8a33-d93fb827debc/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30001919-517f-4031-9ef4-bc0fbb43d122/volumes/kubernetes.io~projected/kube-api-access-4tnhb DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dacad254-b0c1-48fd-8377-1067d65ea230/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/774df0d8e22e6bacf27357b34b44bd971a3a2680b7bcb2bfd028b632d778ea35/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b174c80c-af6a-4f17-8442-1a029b286011/volumes/kubernetes.io~projected/kube-api-access-6w2qd DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d210d1881661386240975f06970c0bb1710c5358b3d15c73a1eb91b9f52dcbcf/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b516f174-836e-4711-8999-80ce5a592e28/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3c7cd045ffa4e727bde3156153e89aa8677f1e94e56a90ceec2ba11980727bf/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/b516f174-836e-4711-8999-80ce5a592e28/volumes/kubernetes.io~projected/kube-api-access-q6zs9 DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbe6588c13763b48fd28430b4f4c5429731f4c47e52fee7e9a79177f59fcfbd0/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3794a50e-6c2f-40bc-9be7-eda1388f3178/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b166ba605a74b057cd8ab0032f9e35754ce05745876024b528a0b101fba0e4c/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:03ebb684cd0cb48 MacAddress:d2:29:ee:f1:79:43 Speed:10000 Mtu:8901} {Name:0b166ba605a74b0 MacAddress:52:22:78:e5:ca:d3 Speed:10000 Mtu:8901} {Name:1da9fa642e9eaa3 MacAddress:5e:04:39:fb:ea:4a Speed:10000 Mtu:8901} {Name:1e59114d4ba6e95 MacAddress:1a:5a:14:69:7e:1d Speed:10000 Mtu:8901} {Name:774df0d8e22e6ba MacAddress:72:13:25:9b:ed:aa Speed:10000 Mtu:8901} {Name:77affc7483735aa MacAddress:da:71:d4:97:26:74 Speed:10000 Mtu:8901} {Name:a196d593fd53d6c MacAddress:46:d5:07:4c:bf:ec Speed:10000 Mtu:8901} {Name:a3cc7e5082a0f48 MacAddress:b6:a9:f4:e1:ff:d7 Speed:10000 Mtu:8901} {Name:b3c7cd045ffa4e7 MacAddress:26:05:0e:25:4c:60 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d9:80:b5:df Speed:0 Mtu:9001} {Name:br-int MacAddress:8e:88:09:e4:63:42 Speed:0 Mtu:8901} {Name:dbe6588c13763b4 MacAddress:8e:d3:3b:e3:95:51 Speed:10000 Mtu:8901} {Name:ecbf1d255616640 MacAddress:ee:cd:4e:23:60:e6 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d9:80:b5:df Speed:0 Mtu:9001} {Name:f18612a92e2dc29 MacAddress:8a:bd:4e:f0:ef:3c Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:2e:af:d0:8b:9b:9f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c6:13:eb:24:83:01 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:47:02.435633 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.435299 4828 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:47:02.435633 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.435348 4828 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:47:02.435851 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.435705 4828 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:47:02.435882 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.435734 4828 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-153.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:47:02.435882 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.435864 4828 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:47:02.435882 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.435872 4828 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:47:02.435981 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.435883 4828 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:47:02.435981 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.435909 4828 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:47:02.435981 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.435931 4828 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:47:02.436128 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.436117 4828 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:47:02.436262 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.436253 4828 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:47:02.436293 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.436288 4828 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:47:02.436320 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.436299 4828 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:47:02.436320 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.436316 4828 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:47:02.436370 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.436325 4828 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:47:02.436370 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.436333 4828 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:47:02.437252 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.437238 4828 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:47:02.437252 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.437251 4828 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:47:02.437751 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.437732 4828 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:47:02.439879 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.439865 4828 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:47:02.439879 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.439879 4828 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:47:02.439990 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.439979 4828 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 16:47:02.440174 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440163 4828 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:47:02.440207 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440192 4828 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:47:02.440285 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440276 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:47:02.440315 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440290 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:47:02.440315 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440296 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:47:02.440315 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440302 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:47:02.440315 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440308 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:47:02.440315 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440314 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:47:02.440449 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440325 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:47:02.440449 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440331 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:47:02.440449 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440341 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:47:02.440449 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440347 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:47:02.440449 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440363 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:47:02.440449 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440372 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:47:02.440449 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440395 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:47:02.440449 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.440401 4828 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:47:02.443769 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.443756 4828 server.go:1263] "Started kubelet" Oct 30 16:47:02.443855 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.443803 4828 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:47:02.444328 ip-10-0-135-153 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:47:02.447639 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.443867 4828 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:47:02.447740 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.447678 4828 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:47:02.448044 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.448028 4828 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:47:02.448676 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.448662 4828 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:47:02.450933 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.449978 4828 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:47:02.450933 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.449997 4828 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:47:02.450933 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.450043 4828 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:47:02.450933 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.450056 4828 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:47:02.450933 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:02.450082 4828 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-153.ec2.internal\" not found" Oct 30 16:47:02.450933 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.450142 4828 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:47:02.450933 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.450046 4828 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:41:23 +0000 UTC" deadline="2028-04-03 12:50:34.140052552 +0000 UTC" Oct 30 16:47:02.450933 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.450175 4828 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12500h3m31.689897665s" Oct 30 16:47:02.453004 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.452955 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68321123-f8fc-4c6a-bd22-1f614a8a3b55" volumeName="kubernetes.io/secret/68321123-f8fc-4c6a-bd22-1f614a8a3b55-cert" seLinuxMountContext="" Oct 30 16:47:02.453069 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453009 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a" volumeName="kubernetes.io/configmap/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a-konnectivity-ca" seLinuxMountContext="" Oct 30 16:47:02.453069 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453025 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b516f174-836e-4711-8999-80ce5a592e28" volumeName="kubernetes.io/configmap/b516f174-836e-4711-8999-80ce5a592e28-trusted-ca-bundle" seLinuxMountContext="" Oct 30 16:47:02.453069 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453040 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b54c34b7-e8c3-4fe9-9b66-36336dfae405" volumeName="kubernetes.io/configmap/b54c34b7-e8c3-4fe9-9b66-36336dfae405-cni-binary-copy" seLinuxMountContext="" Oct 30 16:47:02.453069 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453053 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f" volumeName="kubernetes.io/projected/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-kube-api-access-rvwhw" seLinuxMountContext="" Oct 30 16:47:02.453189 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453067 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3794a50e-6c2f-40bc-9be7-eda1388f3178" volumeName="kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-bound-sa-token" seLinuxMountContext="" Oct 30 16:47:02.453189 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453080 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7918c4b3-b4f4-4435-9de9-77fbe4906a7f" volumeName="kubernetes.io/configmap/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-nginx-conf" seLinuxMountContext="" Oct 30 16:47:02.453189 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453094 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="852f74bb-c91e-4075-8335-aeba3e723709" volumeName="kubernetes.io/secret/852f74bb-c91e-4075-8335-aeba3e723709-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 16:47:02.453189 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453109 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="883b14e2-b8bc-4650-a638-c33673683a45" volumeName="kubernetes.io/projected/883b14e2-b8bc-4650-a638-c33673683a45-kube-api-access-fwc77" seLinuxMountContext="" Oct 30 16:47:02.453189 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453123 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6361ea7-1d84-4bc8-b439-3f60ed9d9e40" volumeName="kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-default-certificate" seLinuxMountContext="" Oct 30 16:47:02.453189 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453136 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b54c34b7-e8c3-4fe9-9b66-36336dfae405" volumeName="kubernetes.io/projected/b54c34b7-e8c3-4fe9-9b66-36336dfae405-kube-api-access-4l2kn" seLinuxMountContext="" Oct 30 16:47:02.453189 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453151 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d38312b4-bd6e-4a5f-8a33-d93fb827debc" volumeName="kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-certificates" seLinuxMountContext="" Oct 30 16:47:02.453189 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453163 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3794a50e-6c2f-40bc-9be7-eda1388f3178" volumeName="kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-certificates" seLinuxMountContext="" Oct 30 16:47:02.453189 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453178 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c30c30a-8c1f-4552-b2d1-2692bc423d45" volumeName="kubernetes.io/projected/8c30c30a-8c1f-4552-b2d1-2692bc423d45-kube-api-access-9nttq" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453194 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b516f174-836e-4711-8999-80ce5a592e28" volumeName="kubernetes.io/configmap/b516f174-836e-4711-8999-80ce5a592e28-service-ca-bundle" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453213 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d38312b4-bd6e-4a5f-8a33-d93fb827debc" volumeName="kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-trusted-ca" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453225 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d38312b4-bd6e-4a5f-8a33-d93fb827debc" volumeName="kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-bound-sa-token" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453238 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f13fd623-29b9-4d8b-b76d-668ca123010f" volumeName="kubernetes.io/configmap/f13fd623-29b9-4d8b-b76d-668ca123010f-config-volume" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453251 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="852f74bb-c91e-4075-8335-aeba3e723709" volumeName="kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-env-overrides" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453262 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="159cde9b-22a5-4930-9bf6-2535ab4af3d4" volumeName="kubernetes.io/projected/159cde9b-22a5-4930-9bf6-2535ab4af3d4-kube-api-access-p4x9q" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453273 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35a14012-73eb-4f3b-b762-f56c6a963e44" volumeName="kubernetes.io/projected/35a14012-73eb-4f3b-b762-f56c6a963e44-kube-api-access-sgkdb" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453284 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d38312b4-bd6e-4a5f-8a33-d93fb827debc" volumeName="kubernetes.io/empty-dir/d38312b4-bd6e-4a5f-8a33-d93fb827debc-ca-trust-extracted" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453295 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbb5e2c6-44f4-4152-a7ac-98ae4c045190" volumeName="kubernetes.io/configmap/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-telemetry-config" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453306 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f13fd623-29b9-4d8b-b76d-668ca123010f" volumeName="kubernetes.io/projected/f13fd623-29b9-4d8b-b76d-668ca123010f-kube-api-access-sqff6" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453317 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3794a50e-6c2f-40bc-9be7-eda1388f3178" volumeName="kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-image-registry-private-configuration" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453329 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a41c35af-f159-4802-9b53-fbb8cbee2104" volumeName="kubernetes.io/projected/a41c35af-f159-4802-9b53-fbb8cbee2104-kube-api-access-rvsrx" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453340 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e" volumeName="kubernetes.io/secret/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-serving-cert" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453351 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b516f174-836e-4711-8999-80ce5a592e28" volumeName="kubernetes.io/empty-dir/b516f174-836e-4711-8999-80ce5a592e28-tmp" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453366 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b516f174-836e-4711-8999-80ce5a592e28" volumeName="kubernetes.io/projected/b516f174-836e-4711-8999-80ce5a592e28-kube-api-access-q6zs9" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453379 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ca4c236-e28d-4331-9336-63a71d706ef0" volumeName="kubernetes.io/projected/3ca4c236-e28d-4331-9336-63a71d706ef0-kube-api-access-svsvb" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453390 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c30c30a-8c1f-4552-b2d1-2692bc423d45" volumeName="kubernetes.io/empty-dir/8c30c30a-8c1f-4552-b2d1-2692bc423d45-tmp" seLinuxMountContext="" Oct 30 16:47:02.453416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453413 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a" volumeName="kubernetes.io/secret/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a-agent-certs" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453425 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a41c35af-f159-4802-9b53-fbb8cbee2104" volumeName="kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-cni-binary-copy" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453436 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b34094c9-7349-4284-8b86-2253f9e4d87f" volumeName="kubernetes.io/configmap/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-config" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453448 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d38312b4-bd6e-4a5f-8a33-d93fb827debc" volumeName="kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-kube-api-access-q52w4" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453459 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d38312b4-bd6e-4a5f-8a33-d93fb827debc" volumeName="kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-image-registry-private-configuration" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453470 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35a14012-73eb-4f3b-b762-f56c6a963e44" volumeName="kubernetes.io/secret/35a14012-73eb-4f3b-b762-f56c6a963e44-serving-cert" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453487 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cdefc81-0135-4053-bc7b-78958686e8ea" volumeName="kubernetes.io/projected/3cdefc81-0135-4053-bc7b-78958686e8ea-kube-api-access-64l24" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453502 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac8ed89d-9808-4bff-9d24-3e5fc34041ad" volumeName="kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453515 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f" volumeName="kubernetes.io/configmap/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-serviceca" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453525 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35a14012-73eb-4f3b-b762-f56c6a963e44" volumeName="kubernetes.io/configmap/35a14012-73eb-4f3b-b762-f56c6a963e44-config" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453538 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3794a50e-6c2f-40bc-9be7-eda1388f3178" volumeName="kubernetes.io/empty-dir/3794a50e-6c2f-40bc-9be7-eda1388f3178-ca-trust-extracted" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453550 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3794a50e-6c2f-40bc-9be7-eda1388f3178" volumeName="kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-installation-pull-secrets" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453562 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b34094c9-7349-4284-8b86-2253f9e4d87f" volumeName="kubernetes.io/projected/b34094c9-7349-4284-8b86-2253f9e4d87f-kube-api-access-zmgvx" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453574 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dacad254-b0c1-48fd-8377-1067d65ea230" volumeName="kubernetes.io/configmap/dacad254-b0c1-48fd-8377-1067d65ea230-trusted-ca" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453585 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="852f74bb-c91e-4075-8335-aeba3e723709" volumeName="kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-ovnkube-config" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453596 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6361ea7-1d84-4bc8-b439-3f60ed9d9e40" volumeName="kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-stats-auth" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453607 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b174c80c-af6a-4f17-8442-1a029b286011" volumeName="kubernetes.io/projected/b174c80c-af6a-4f17-8442-1a029b286011-kube-api-access-6w2qd" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453620 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b516f174-836e-4711-8999-80ce5a592e28" volumeName="kubernetes.io/empty-dir/b516f174-836e-4711-8999-80ce5a592e28-snapshots" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453632 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aeb454-5336-4fc1-8194-54d54a6cdb6c" volumeName="kubernetes.io/empty-dir/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-tmp-dir" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453644 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="852f74bb-c91e-4075-8335-aeba3e723709" volumeName="kubernetes.io/projected/852f74bb-c91e-4075-8335-aeba3e723709-kube-api-access-vp752" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453655 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b54c34b7-e8c3-4fe9-9b66-36336dfae405" volumeName="kubernetes.io/configmap/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-daemon-config" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453674 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d38312b4-bd6e-4a5f-8a33-d93fb827debc" volumeName="kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-installation-pull-secrets" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453689 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dacad254-b0c1-48fd-8377-1067d65ea230" volumeName="kubernetes.io/projected/dacad254-b0c1-48fd-8377-1067d65ea230-kube-api-access-8cmkh" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453701 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="883b14e2-b8bc-4650-a638-c33673683a45" volumeName="kubernetes.io/empty-dir/883b14e2-b8bc-4650-a638-c33673683a45-data-volume" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453713 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e" volumeName="kubernetes.io/configmap/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-config" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453753 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3794a50e-6c2f-40bc-9be7-eda1388f3178" volumeName="kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-kube-api-access-7xjxv" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453765 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a132686f-9475-437d-826e-22fb93e86323" volumeName="kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453778 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dacad254-b0c1-48fd-8377-1067d65ea230" volumeName="kubernetes.io/secret/dacad254-b0c1-48fd-8377-1067d65ea230-serving-cert" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453790 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f13fd623-29b9-4d8b-b76d-668ca123010f" volumeName="kubernetes.io/empty-dir/f13fd623-29b9-4d8b-b76d-668ca123010f-tmp-dir" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453802 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="252c7435-f4c1-4dac-a468-e3287c6a7e9e" volumeName="kubernetes.io/projected/252c7435-f4c1-4dac-a468-e3287c6a7e9e-kube-api-access-774sr" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453823 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a04182-8031-4fa8-8bed-4f8b60b6cc97" volumeName="kubernetes.io/projected/d7a04182-8031-4fa8-8bed-4f8b60b6cc97-kube-api-access-4srgz" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453834 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dacad254-b0c1-48fd-8377-1067d65ea230" volumeName="kubernetes.io/configmap/dacad254-b0c1-48fd-8377-1067d65ea230-config" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453845 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbb5e2c6-44f4-4152-a7ac-98ae4c045190" volumeName="kubernetes.io/projected/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-kube-api-access-gm5w2" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453855 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3095c033-c4ff-42cd-9530-1be17a144a31" volumeName="kubernetes.io/projected/3095c033-c4ff-42cd-9530-1be17a144a31-kube-api-access-fgpkr" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453868 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8aeb454-5336-4fc1-8194-54d54a6cdb6c" volumeName="kubernetes.io/projected/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-kube-api-access-72lmm" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453879 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a41c35af-f159-4802-9b53-fbb8cbee2104" volumeName="kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453890 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a41c35af-f159-4802-9b53-fbb8cbee2104" volumeName="kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453920 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b174c80c-af6a-4f17-8442-1a029b286011" volumeName="kubernetes.io/configmap/b174c80c-af6a-4f17-8442-1a029b286011-iptables-alerter-script" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453931 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30001919-517f-4031-9ef4-bc0fbb43d122" volumeName="kubernetes.io/projected/30001919-517f-4031-9ef4-bc0fbb43d122-kube-api-access-4tnhb" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453945 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68321123-f8fc-4c6a-bd22-1f614a8a3b55" volumeName="kubernetes.io/projected/68321123-f8fc-4c6a-bd22-1f614a8a3b55-kube-api-access-qprsq" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453957 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c30c30a-8c1f-4552-b2d1-2692bc423d45" volumeName="kubernetes.io/empty-dir/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-tuned" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453969 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6361ea7-1d84-4bc8-b439-3f60ed9d9e40" volumeName="kubernetes.io/projected/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-kube-api-access-pjrlc" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453981 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b516f174-836e-4711-8999-80ce5a592e28" volumeName="kubernetes.io/secret/b516f174-836e-4711-8999-80ce5a592e28-serving-cert" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.453992 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="852f74bb-c91e-4075-8335-aeba3e723709" volumeName="kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-ovnkube-script-lib" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.454003 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e" volumeName="kubernetes.io/projected/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-kube-api-access-z8f6w" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.454015 4828 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3794a50e-6c2f-40bc-9be7-eda1388f3178" volumeName="kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-trusted-ca" seLinuxMountContext="" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.454039 4828 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.454046 4828 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.454116 4828 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.454130 4828 factory.go:55] Registering systemd factory Oct 30 16:47:02.454162 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.454140 4828 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:47:02.458196 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.458079 4828 factory.go:153] Registering CRI-O factory Oct 30 16:47:02.459058 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.458100 4828 factory.go:223] Registration of the crio container factory successfully Oct 30 16:47:02.459147 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.459072 4828 factory.go:103] Registering Raw factory Oct 30 16:47:02.459147 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.459085 4828 manager.go:1199] Started watching for new ooms in manager Oct 30 16:47:02.459450 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.459433 4828 manager.go:319] Starting recovery of all containers Oct 30 16:47:02.467625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.467592 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/1.log" Oct 30 16:47:02.467765 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.467678 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/1.log" Oct 30 16:47:02.480378 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.480065 4828 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:47:02.481651 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.481624 4828 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:47:02.481651 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.481649 4828 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:47:02.481813 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.481676 4828 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:47:02.481813 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:02.481720 4828 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 16:47:02.498774 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.498744 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/1.log" Oct 30 16:47:02.498935 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.498782 4828 generic.go:356] "Generic (PLEG): container finished" podID="dacad254-b0c1-48fd-8377-1067d65ea230" containerID="c99e50eea3562218c0bb5c17d257f8fbe287413615ed42c5c59b5e6a7a33f71f" exitCode=255 Oct 30 16:47:02.502171 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.502146 4828 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 16:47:02.503049 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.503029 4828 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:47:02.503049 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.503046 4828 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:47:02.514423 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.514396 4828 generic.go:356] "Generic (PLEG): container finished" podID="a41c35af-f159-4802-9b53-fbb8cbee2104" containerID="1e1ad57e69290cb7a4aaf9f2cc0733e63f72524c478c3f7622be0e0d2ea15ac1" exitCode=0 Oct 30 16:47:02.514423 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.514419 4828 generic.go:356] "Generic (PLEG): container finished" podID="a41c35af-f159-4802-9b53-fbb8cbee2104" containerID="7425163bdce5c2faa90c91f3aab24425b8adc382d0729316790af2edbcfea31f" exitCode=0 Oct 30 16:47:02.514423 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.514428 4828 generic.go:356] "Generic (PLEG): container finished" podID="a41c35af-f159-4802-9b53-fbb8cbee2104" containerID="93c5cc87c39b5c64f18b6e6a18eb092243d33bb1e4c8c8206b63cac1a91ec277" exitCode=0 Oct 30 16:47:02.514664 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.514436 4828 generic.go:356] "Generic (PLEG): container finished" podID="a41c35af-f159-4802-9b53-fbb8cbee2104" containerID="38d3745e75913fcefb6e44b822e1ff32e70e6c1e75c2ff0280de9c183ec57a8b" exitCode=0 Oct 30 16:47:02.514664 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.514442 4828 generic.go:356] "Generic (PLEG): container finished" podID="a41c35af-f159-4802-9b53-fbb8cbee2104" containerID="6d32e95e901f83fd7efd2343a32e18c5cea99a687d713c566510ef38715b5ab8" exitCode=0 Oct 30 16:47:02.514664 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.514447 4828 generic.go:356] "Generic (PLEG): container finished" podID="a41c35af-f159-4802-9b53-fbb8cbee2104" containerID="d18abfe4e19250e811e5f2e22d697b907716050fbc4a97e26d78c151866a6b34" exitCode=0 Oct 30 16:47:02.516494 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.516475 4828 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:47:02.516617 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.516492 4828 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:47:02.518223 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.518199 4828 generic.go:356] "Generic (PLEG): container finished" podID="b9b4913d9c6bf71319283ff1eade5a9e" containerID="d6323d2c4eb8c66f04c004580ed9ce2b3a678b1400e289509e24873faf812caf" exitCode=0 Oct 30 16:47:02.548658 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.548594 4828 manager.go:324] Recovery completed Oct 30 16:47:02.568048 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.568023 4828 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:47:02.568048 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.568042 4828 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:47:02.568222 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.568066 4828 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:47:02.568222 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.568194 4828 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 16:47:02.568222 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.568207 4828 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 16:47:02.568337 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.568224 4828 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 16:47:02.568337 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.568230 4828 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 16:47:02.568337 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.568235 4828 policy_none.go:50] "Start" Oct 30 16:47:02.568337 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.568279 4828 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:47:02.568337 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.568290 4828 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:47:02.568482 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.568433 4828 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 16:47:02.568482 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.568440 4828 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 16:47:02.568482 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.568446 4828 policy_none.go:44] "Start" Oct 30 16:47:02.572746 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.572713 4828 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:47:02.577889 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.577865 4828 manager.go:341] "Starting Device Plugin manager" Oct 30 16:47:02.578353 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:02.577910 4828 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:47:02.578412 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.578356 4828 server.go:87] "Starting device plugin registration server" Oct 30 16:47:02.578617 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.578604 4828 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:47:02.578651 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.578618 4828 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:47:02.578732 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.578714 4828 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:47:02.578871 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.578829 4828 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:47:02.578979 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.578877 4828 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:47:02.578979 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.578885 4828 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:47:02.579086 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.578933 4828 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:47:02.578865851Z","UUID":"2b51819d-cd62-419e-a6f3-081c289f62f3","Handler":null,"Name":"","Endpoint":""} Oct 30 16:47:02.580560 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:02.580505 4828 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:47:02.580673 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.580640 4828 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:47:02.580673 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.580669 4828 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:47:02.582150 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.581960 4828 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal"] Oct 30 16:47:02.583300 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.583280 4828 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.583421 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.583405 4828 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.583525 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.583278 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" event={"ID":"b9b4913d9c6bf71319283ff1eade5a9e","Type":"ContainerStarted","Data":"af1e79e51e006f8e49c76b10060e39d9fdcb81b1f28a0648cc795e0c97928c03"} Oct 30 16:47:02.583647 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.583629 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" event={"ID":"b9b4913d9c6bf71319283ff1eade5a9e","Type":"ContainerDied","Data":"d6323d2c4eb8c66f04c004580ed9ce2b3a678b1400e289509e24873faf812caf"} Oct 30 16:47:02.583772 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.583758 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" event={"ID":"b9b4913d9c6bf71319283ff1eade5a9e","Type":"ContainerStarted","Data":"3d9a4c93cd8726956f1a8f7901706b5a824a9ce074b05b8adb54875dd8ecd2bd"} Oct 30 16:47:02.584205 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.584166 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" event={"ID":"8216a0ebd9e778d471b8e472a27d4665","Type":"ContainerStarted","Data":"3425f923fe6b8c35b092bec775da87167619792193331d8d26f6d96c69a3ef78"} Oct 30 16:47:02.584295 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.584215 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" event={"ID":"8216a0ebd9e778d471b8e472a27d4665","Type":"ContainerStarted","Data":"d7120dbf5c69333bd3e5babceeda19cf41131e73908d132cd7c31a3b7b882137"} Oct 30 16:47:02.611265 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.611237 4828 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:47:02.611415 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:02.611302 4828 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.611962 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.611944 4828 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:47:02.612040 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:02.611991 4828 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-153.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.679577 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.679546 4828 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:47:02.680851 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.680834 4828 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:47:02.680962 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.680864 4828 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:47:02.680962 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.680874 4828 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-153.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:47:02.680962 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.680933 4828 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.702399 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.702367 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9b4913d9c6bf71319283ff1eade5a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" (UID: \"b9b4913d9c6bf71319283ff1eade5a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.702399 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.702400 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9b4913d9c6bf71319283ff1eade5a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" (UID: \"b9b4913d9c6bf71319283ff1eade5a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.702570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.702421 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8216a0ebd9e778d471b8e472a27d4665-config\") pod \"kube-apiserver-proxy-ip-10-0-135-153.ec2.internal\" (UID: \"8216a0ebd9e778d471b8e472a27d4665\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.717915 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.717873 4828 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.718005 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.717972 4828 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.802924 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.802825 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9b4913d9c6bf71319283ff1eade5a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" (UID: \"b9b4913d9c6bf71319283ff1eade5a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.802924 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.802846 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9b4913d9c6bf71319283ff1eade5a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" (UID: \"b9b4913d9c6bf71319283ff1eade5a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.802924 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.802876 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8216a0ebd9e778d471b8e472a27d4665-config\") pod \"kube-apiserver-proxy-ip-10-0-135-153.ec2.internal\" (UID: \"8216a0ebd9e778d471b8e472a27d4665\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.802924 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.802914 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9b4913d9c6bf71319283ff1eade5a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" (UID: \"b9b4913d9c6bf71319283ff1eade5a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.803146 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.802938 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8216a0ebd9e778d471b8e472a27d4665-config\") pod \"kube-apiserver-proxy-ip-10-0-135-153.ec2.internal\" (UID: \"8216a0ebd9e778d471b8e472a27d4665\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" Oct 30 16:47:02.803146 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:02.802971 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b9b4913d9c6bf71319283ff1eade5a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" (UID: \"b9b4913d9c6bf71319283ff1eade5a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:47:03.437444 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.437409 4828 apiserver.go:52] "Watching apiserver" Oct 30 16:47:03.510187 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.510147 4828 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:47:03.511015 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.510985 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-l4rsf","kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal","openshift-insights/insights-runtime-extractor-j7sxm","openshift-network-console/networking-console-plugin-86c478bb88-7zrs4","kube-system/konnectivity-agent-4hlmr","openshift-multus/multus-q9f8r","openshift-ingress-canary/ingress-canary-tb2dr","openshift-console-operator/console-operator-7cd56b8dbc-n758f","openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2","openshift-multus/network-metrics-daemon-8mpgm","openshift-image-registry/image-registry-67785c878-8c9pr","kube-system/global-pull-secret-syncer-wgxv6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq","kube-system/kas-connection-checker-685895c7c9-mmhr5","openshift-cluster-node-tuning-operator/tuned-c7ckl","openshift-image-registry/image-registry-797d7cb6cf-rzmrh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal","kube-system/kas-connection-checker-685895c7c9-mhsj5","openshift-dns/dns-default-6rrfj","openshift-ovn-kubernetes/ovnkube-node-2ljft","openshift-multus/multus-additional-cni-plugins-2hh98","openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5","openshift-insights/insights-operator-7c4986d449-jmx89","openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52","kube-system/kas-connection-checker-685895c7c9-fb8qf","openshift-dns/node-resolver-nbbqf","openshift-image-registry/node-ca-hfsmw","openshift-ingress/router-default-5c78887cdf-9jhzv","openshift-network-diagnostics/network-check-target-scdcl"] Oct 30 16:47:03.515045 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.515022 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:03.515155 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.515135 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.516863 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.516840 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:47:03.517324 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.517297 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:47:03.518681 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.518663 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:47:03.520976 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.520957 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:47:03.524484 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.524467 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.525456 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.525443 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:03.525975 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.525958 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:03.528174 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.528156 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.528240 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.528190 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.530718 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.530703 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 16:47:03.530766 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.530742 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.530818 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.530797 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.530852 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.530825 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.531086 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.531073 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 16:47:03.531307 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.531277 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 16:47:03.531307 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.531294 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mcfkl\"" Oct 30 16:47:03.531307 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.531308 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 16:47:03.533567 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.533551 4828 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" Oct 30 16:47:03.533701 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.533685 4828 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:47:03.550520 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.550498 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 16:47:03.550834 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.550816 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 16:47:03.550937 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.550886 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nwhz8\"" Oct 30 16:47:03.552071 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.552056 4828 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:47:03.555207 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.555189 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 16:47:03.555314 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.555300 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:47:03.555376 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.555340 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 16:47:03.555878 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.555824 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 16:47:03.555955 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.555881 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2twsj\"" Oct 30 16:47:03.556006 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.555957 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zs99n\"" Oct 30 16:47:03.556389 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.556366 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-c2bll\"" Oct 30 16:47:03.556505 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.556443 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-b86vs\"" Oct 30 16:47:03.556573 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.556515 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ffzh4\"" Oct 30 16:47:03.565276 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.565250 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 16:47:03.565398 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.565308 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:47:03.565398 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.565307 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.565398 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.565336 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.575439 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.575416 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.575802 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.575780 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:47:03.575980 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.575965 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:47:03.577024 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.576430 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.577024 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.576580 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.577024 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.576588 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.577199 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.577166 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:47:03.577456 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.577437 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 16:47:03.585062 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.585043 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.589557 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.589531 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 16:47:03.589655 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.589597 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 16:47:03.589721 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.589653 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.589721 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.589704 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:47:03.589796 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.589767 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 16:47:03.589876 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.589846 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.589978 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.589963 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 16:47:03.590049 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.589992 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8hr2j\"" Oct 30 16:47:03.590049 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590030 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 16:47:03.590139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590091 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:47:03.590139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590105 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-76ckj\"" Oct 30 16:47:03.590139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590127 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8fhb8\"" Oct 30 16:47:03.590281 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590148 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:47:03.590281 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590171 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-8568q\"" Oct 30 16:47:03.590281 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590177 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bdrw\"" Oct 30 16:47:03.590281 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590220 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:47:03.590281 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590241 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pdblj\"" Oct 30 16:47:03.590281 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590247 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:47:03.590281 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590244 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fxkbf\"" Oct 30 16:47:03.590553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590348 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.590553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590357 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:47:03.590553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590426 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7stlv\"" Oct 30 16:47:03.590553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590455 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pjmpr\"" Oct 30 16:47:03.590553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590460 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6bljg\"" Oct 30 16:47:03.590553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590475 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.590553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590493 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 16:47:03.590553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590500 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.590553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590510 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.590553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590519 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.590553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590528 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.590553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590553 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590564 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590571 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590576 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z7xnd\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590581 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590576 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590610 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590588 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590615 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590594 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590673 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590696 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590757 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9wpv\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590818 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2qtkg\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590885 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rbwcw\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590927 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wd4nn\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.590960 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-szhrj\"" Oct 30 16:47:03.591012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.591023 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j4x52\"" Oct 30 16:47:03.591579 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.591069 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7c68\"" Oct 30 16:47:03.591579 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.591113 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.591579 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.591151 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.591579 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.591220 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.591579 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.591242 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.591579 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.591288 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:47:03.591579 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.591323 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:47:03.594813 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.594790 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.598210 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.598197 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:47:03.602258 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.602243 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:47:03.602258 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.602254 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 16:47:03.602368 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.602271 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:47:03.602368 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.602296 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.602368 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.602313 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 16:47:03.602368 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.602333 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-g8gt8\"" Oct 30 16:47:03.602368 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.602338 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:47:03.602368 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.602334 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:47:03.602558 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.602385 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:47:03.602558 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.602437 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:47:03.603942 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.603926 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:47:03.603942 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.603939 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 16:47:03.604039 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.603949 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 16:47:03.604301 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.604287 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 16:47:03.606061 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606046 4828 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:47:03.606148 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.606106 4828 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-153.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-153.ec2.internal" Oct 30 16:47:03.606224 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606206 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nttq\" (UniqueName: \"kubernetes.io/projected/8c30c30a-8c1f-4552-b2d1-2692bc423d45-kube-api-access-9nttq\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.606272 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606236 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-run-netns\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.606272 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606255 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c30c30a-8c1f-4552-b2d1-2692bc423d45-tmp\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.606354 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606273 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:47:03.606354 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606322 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svsvb\" (UniqueName: \"kubernetes.io/projected/3ca4c236-e28d-4331-9336-63a71d706ef0-kube-api-access-svsvb\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:47:03.606446 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606368 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c30c30a-8c1f-4552-b2d1-2692bc423d45-tmp\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.606446 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606406 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-systemd-units\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.606532 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606517 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-env-overrides\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.606583 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606566 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-etc-kubernetes\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.606658 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606641 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-cni-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.606715 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606685 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysctl-conf\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.606814 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606798 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:47:03.606848 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606836 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-default-certificate\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:03.606877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606846 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-env-overrides\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.606939 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606925 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-netns\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.606980 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.606964 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-cni-bin\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.607030 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607003 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b516f174-836e-4711-8999-80ce5a592e28-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:47:03.607067 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607033 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6zs9\" (UniqueName: \"kubernetes.io/projected/b516f174-836e-4711-8999-80ce5a592e28-kube-api-access-q6zs9\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:47:03.607067 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607040 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-default-certificate\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:03.607157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607063 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b174c80c-af6a-4f17-8442-1a029b286011-iptables-alerter-script\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:47:03.607157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607070 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-nginx-conf\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:47:03.607157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607151 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-socket-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.607283 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607172 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-kubelet\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.607283 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607188 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-node-log\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.607283 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607214 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-socket-dir-parent\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.607283 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607260 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-k8s-cni-cncf-io\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.607456 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607295 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgkdb\" (UniqueName: \"kubernetes.io/projected/35a14012-73eb-4f3b-b762-f56c6a963e44-kube-api-access-sgkdb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:47:03.607456 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607327 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64l24\" (UniqueName: \"kubernetes.io/projected/3cdefc81-0135-4053-bc7b-78958686e8ea-kube-api-access-64l24\") pod \"network-check-source-6f78d59f99-jdsr5\" (UID: \"3cdefc81-0135-4053-bc7b-78958686e8ea\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5" Oct 30 16:47:03.607456 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607369 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:47:03.607456 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607433 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b174c80c-af6a-4f17-8442-1a029b286011-iptables-alerter-script\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:47:03.607456 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607440 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b516f174-836e-4711-8999-80ce5a592e28-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:47:03.607649 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607463 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-cni-multus\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.607649 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607551 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgpkr\" (UniqueName: \"kubernetes.io/projected/3095c033-c4ff-42cd-9530-1be17a144a31-kube-api-access-fgpkr\") pod \"kas-connection-checker-685895c7c9-fb8qf\" (UID: \"3095c033-c4ff-42cd-9530-1be17a144a31\") " pod="kube-system/kas-connection-checker-685895c7c9-fb8qf" Oct 30 16:47:03.607649 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607575 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-original-pull-secret\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:47:03.607649 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607581 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-certificates\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.607649 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607599 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-dbus\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:47:03.607649 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607624 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b54c34b7-e8c3-4fe9-9b66-36336dfae405-cni-binary-copy\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.607649 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607639 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-daemon-config\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.607949 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607654 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm5w2\" (UniqueName: \"kubernetes.io/projected/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-kube-api-access-gm5w2\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:47:03.607949 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607689 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:03.607949 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607742 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-certificates\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.607949 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607789 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-modprobe-d\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.607949 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607815 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a-konnectivity-ca\") pod \"konnectivity-agent-4hlmr\" (UID: \"9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a\") " pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:47:03.607949 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607838 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.607949 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607862 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-tuned\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.607949 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607929 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-daemon-config\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.607949 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607936 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2qd\" (UniqueName: \"kubernetes.io/projected/b174c80c-af6a-4f17-8442-1a029b286011-kube-api-access-6w2qd\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:47:03.607949 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607950 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-tuned\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607953 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b54c34b7-e8c3-4fe9-9b66-36336dfae405-cni-binary-copy\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.607987 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnhb\" (UniqueName: \"kubernetes.io/projected/30001919-517f-4031-9ef4-bc0fbb43d122-kube-api-access-4tnhb\") pod \"kas-connection-checker-685895c7c9-mmhr5\" (UID: \"30001919-517f-4031-9ef4-bc0fbb43d122\") " pod="kube-system/kas-connection-checker-685895c7c9-mmhr5" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608014 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f13fd623-29b9-4d8b-b76d-668ca123010f-config-volume\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608038 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-image-registry-private-configuration\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608066 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-trusted-ca\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608094 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dacad254-b0c1-48fd-8377-1067d65ea230-trusted-ca\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608115 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a-konnectivity-ca\") pod \"konnectivity-agent-4hlmr\" (UID: \"9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a\") " pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608119 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-cnibin\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608143 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-hostroot\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608166 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-var-lib-kubelet\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608214 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dacad254-b0c1-48fd-8377-1067d65ea230-serving-cert\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608252 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f13fd623-29b9-4d8b-b76d-668ca123010f-config-volume\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608259 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-image-registry-private-configuration\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608260 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q52w4\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-kube-api-access-q52w4\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608297 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l2kn\" (UniqueName: \"kubernetes.io/projected/b54c34b7-e8c3-4fe9-9b66-36336dfae405-kube-api-access-4l2kn\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608314 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608341 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-cnibin\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608377 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.608380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608388 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dacad254-b0c1-48fd-8377-1067d65ea230-trusted-ca\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608388 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dacad254-b0c1-48fd-8377-1067d65ea230-serving-cert\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608431 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-trusted-ca\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608441 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b516f174-836e-4711-8999-80ce5a592e28-serving-cert\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608461 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608483 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-cni-netd\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608498 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608542 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-kubernetes\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608571 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b516f174-836e-4711-8999-80ce5a592e28-serving-cert\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608585 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-serving-cert\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608633 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608690 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-config\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608709 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608724 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-systemd\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608740 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-ovn\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608742 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-serving-cert\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608789 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4x9q\" (UniqueName: \"kubernetes.io/projected/159cde9b-22a5-4930-9bf6-2535ab4af3d4-kube-api-access-p4x9q\") pod \"kas-connection-checker-685895c7c9-mhsj5\" (UID: \"159cde9b-22a5-4930-9bf6-2535ab4af3d4\") " pod="kube-system/kas-connection-checker-685895c7c9-mhsj5" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608808 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-serviceca\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608824 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8f6w\" (UniqueName: \"kubernetes.io/projected/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-kube-api-access-z8f6w\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608840 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-hosts-file\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.608876 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609087 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:47:03.609190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609096 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-config\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609251 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-installation-pull-secrets\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609272 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srgz\" (UniqueName: \"kubernetes.io/projected/d7a04182-8031-4fa8-8bed-4f8b60b6cc97-kube-api-access-4srgz\") pod \"volume-data-source-validator-58c9f5f46d-ggh56\" (UID: \"d7a04182-8031-4fa8-8bed-4f8b60b6cc97\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609293 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609312 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609313 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-serviceca\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609335 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609379 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/883b14e2-b8bc-4650-a638-c33673683a45-data-volume\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609423 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dacad254-b0c1-48fd-8377-1067d65ea230-config\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609442 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-image-registry-private-configuration\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609454 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-installation-pull-secrets\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609460 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-host\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609482 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609506 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609558 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609593 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609606 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609636 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-device-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609631 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dacad254-b0c1-48fd-8377-1067d65ea230-config\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609664 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/883b14e2-b8bc-4650-a638-c33673683a45-data-volume\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609681 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-sys-fs\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609712 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqff6\" (UniqueName: \"kubernetes.io/projected/f13fd623-29b9-4d8b-b76d-668ca123010f-kube-api-access-sqff6\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609723 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-image-registry-private-configuration\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609765 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b516f174-836e-4711-8999-80ce5a592e28-service-ca-bundle\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609797 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysconfig\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609815 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-sys\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609838 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qprsq\" (UniqueName: \"kubernetes.io/projected/68321123-f8fc-4c6a-bd22-1f614a8a3b55-kube-api-access-qprsq\") pod \"ingress-canary-tb2dr\" (UID: \"68321123-f8fc-4c6a-bd22-1f614a8a3b55\") " pod="openshift-ingress-canary/ingress-canary-tb2dr" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609878 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-slash\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609916 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b516f174-836e-4711-8999-80ce5a592e28-service-ca-bundle\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609930 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-certificates\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609960 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b174c80c-af6a-4f17-8442-1a029b286011-host-slash\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.609987 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvsrx\" (UniqueName: \"kubernetes.io/projected/a41c35af-f159-4802-9b53-fbb8cbee2104-kube-api-access-rvsrx\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610100 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/883b14e2-b8bc-4650-a638-c33673683a45-crio-socket\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610107 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-certificates\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.610139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610154 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f13fd623-29b9-4d8b-b76d-668ca123010f-tmp-dir\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610192 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610221 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35a14012-73eb-4f3b-b762-f56c6a963e44-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610227 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f13fd623-29b9-4d8b-b76d-668ca123010f-tmp-dir\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610262 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-host\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610320 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-run-ovn-kubernetes\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610357 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-kubelet-config\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610390 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-systemd\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610407 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/35a14012-73eb-4f3b-b762-f56c6a963e44-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610426 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610471 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610493 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-ovnkube-config\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610509 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-registration-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610550 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d38312b4-bd6e-4a5f-8a33-d93fb827debc-ca-trust-extracted\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610635 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d38312b4-bd6e-4a5f-8a33-d93fb827debc-ca-trust-extracted\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610673 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-ovnkube-config\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610677 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68321123-f8fc-4c6a-bd22-1f614a8a3b55-cert\") pod \"ingress-canary-tb2dr\" (UID: \"68321123-f8fc-4c6a-bd22-1f614a8a3b55\") " pod="openshift-ingress-canary/ingress-canary-tb2dr" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610796 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp752\" (UniqueName: \"kubernetes.io/projected/852f74bb-c91e-4075-8335-aeba3e723709-kube-api-access-vp752\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610815 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cmkh\" (UniqueName: \"kubernetes.io/projected/dacad254-b0c1-48fd-8377-1067d65ea230-kube-api-access-8cmkh\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610830 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-trusted-ca\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610851 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzz5\" (UniqueName: \"kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5\") pod \"network-check-target-scdcl\" (UID: \"a132686f-9475-437d-826e-22fb93e86323\") " pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610868 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-system-cni-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610877 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/68321123-f8fc-4c6a-bd22-1f614a8a3b55-cert\") pod \"ingress-canary-tb2dr\" (UID: \"68321123-f8fc-4c6a-bd22-1f614a8a3b55\") " pod="openshift-ingress-canary/ingress-canary-tb2dr" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610883 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-conf-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.610917 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xjxv\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-kube-api-access-7xjxv\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611077 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysctl-d\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611110 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-run\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611151 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrlc\" (UniqueName: \"kubernetes.io/projected/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-kube-api-access-pjrlc\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611171 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611187 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-bound-sa-token\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611205 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmgvx\" (UniqueName: \"kubernetes.io/projected/b34094c9-7349-4284-8b86-2253f9e4d87f-kube-api-access-zmgvx\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611291 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-var-lib-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611315 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-etc-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611331 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/852f74bb-c91e-4075-8335-aeba3e723709-ovn-node-metrics-cert\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611355 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwc77\" (UniqueName: \"kubernetes.io/projected/883b14e2-b8bc-4650-a638-c33673683a45-kube-api-access-fwc77\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611377 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-lib-modules\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.611405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611400 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-774sr\" (UniqueName: \"kubernetes.io/projected/252c7435-f4c1-4dac-a468-e3287c6a7e9e-kube-api-access-774sr\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611442 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/852f74bb-c91e-4075-8335-aeba3e723709-ovn-node-metrics-cert\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611443 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a-agent-certs\") pod \"konnectivity-agent-4hlmr\" (UID: \"9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a\") " pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611470 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-log-socket\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611495 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-cni-bin\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611514 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-cni-binary-copy\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611531 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-bound-sa-token\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611556 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35a14012-73eb-4f3b-b762-f56c6a963e44-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611664 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/35a14012-73eb-4f3b-b762-f56c6a963e44-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611697 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a-agent-certs\") pod \"konnectivity-agent-4hlmr\" (UID: \"9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a\") " pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611707 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-trusted-ca\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611726 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a41c35af-f159-4802-9b53-fbb8cbee2104-cni-binary-copy\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611741 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-tmp-dir\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611769 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-system-cni-dir\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611804 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-tmp-dir\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611813 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-kubelet\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611842 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-multus-certs\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611871 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b516f174-836e-4711-8999-80ce5a592e28-snapshots\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611916 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwhw\" (UniqueName: \"kubernetes.io/projected/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-kube-api-access-rvwhw\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611939 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-installation-pull-secrets\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611944 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b516f174-836e-4711-8999-80ce5a592e28-snapshots\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611956 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-ovnkube-script-lib\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611983 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72lmm\" (UniqueName: \"kubernetes.io/projected/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-kube-api-access-72lmm\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.611999 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3794a50e-6c2f-40bc-9be7-eda1388f3178-ca-trust-extracted\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.612066 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3794a50e-6c2f-40bc-9be7-eda1388f3178-ca-trust-extracted\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.612087 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b516f174-836e-4711-8999-80ce5a592e28-tmp\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.612112 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-stats-auth\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.612128 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.612151 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b516f174-836e-4711-8999-80ce5a592e28-tmp\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.612157 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/852f74bb-c91e-4075-8335-aeba3e723709-ovnkube-script-lib\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.612174 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-installation-pull-secrets\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.612678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.612304 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-stats-auth\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:03.615342 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.615328 4828 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:47:03.615436 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.615387 4828 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-153.ec2.internal" Oct 30 16:47:03.635708 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.635680 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6zs9\" (UniqueName: \"kubernetes.io/projected/b516f174-836e-4711-8999-80ce5a592e28-kube-api-access-q6zs9\") pod \"insights-operator-7c4986d449-jmx89\" (UID: \"b516f174-836e-4711-8999-80ce5a592e28\") " pod="openshift-insights/insights-operator-7c4986d449-jmx89" Oct 30 16:47:03.636850 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.636832 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l2kn\" (UniqueName: \"kubernetes.io/projected/b54c34b7-e8c3-4fe9-9b66-36336dfae405-kube-api-access-4l2kn\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.637185 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.637168 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnhb\" (UniqueName: \"kubernetes.io/projected/30001919-517f-4031-9ef4-bc0fbb43d122-kube-api-access-4tnhb\") pod \"kas-connection-checker-685895c7c9-mmhr5\" (UID: \"30001919-517f-4031-9ef4-bc0fbb43d122\") " pod="kube-system/kas-connection-checker-685895c7c9-mmhr5" Oct 30 16:47:03.637853 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.637838 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgpkr\" (UniqueName: \"kubernetes.io/projected/3095c033-c4ff-42cd-9530-1be17a144a31-kube-api-access-fgpkr\") pod \"kas-connection-checker-685895c7c9-fb8qf\" (UID: \"3095c033-c4ff-42cd-9530-1be17a144a31\") " pod="kube-system/kas-connection-checker-685895c7c9-fb8qf" Oct 30 16:47:03.640324 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.640307 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srgz\" (UniqueName: \"kubernetes.io/projected/d7a04182-8031-4fa8-8bed-4f8b60b6cc97-kube-api-access-4srgz\") pod \"volume-data-source-validator-58c9f5f46d-ggh56\" (UID: \"d7a04182-8031-4fa8-8bed-4f8b60b6cc97\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ggh56" Oct 30 16:47:03.640412 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.640397 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64l24\" (UniqueName: \"kubernetes.io/projected/3cdefc81-0135-4053-bc7b-78958686e8ea-kube-api-access-64l24\") pod \"network-check-source-6f78d59f99-jdsr5\" (UID: \"3cdefc81-0135-4053-bc7b-78958686e8ea\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-jdsr5" Oct 30 16:47:03.641518 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.641503 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8f6w\" (UniqueName: \"kubernetes.io/projected/11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e-kube-api-access-z8f6w\") pod \"service-ca-operator-544d749c8c-kzh45\" (UID: \"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" Oct 30 16:47:03.641611 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.641597 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm5w2\" (UniqueName: \"kubernetes.io/projected/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-kube-api-access-gm5w2\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:47:03.641975 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.641961 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgkdb\" (UniqueName: \"kubernetes.io/projected/35a14012-73eb-4f3b-b762-f56c6a963e44-kube-api-access-sgkdb\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq\" (UID: \"35a14012-73eb-4f3b-b762-f56c6a963e44\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" Oct 30 16:47:03.650240 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.650221 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4x9q\" (UniqueName: \"kubernetes.io/projected/159cde9b-22a5-4930-9bf6-2535ab4af3d4-kube-api-access-p4x9q\") pod \"kas-connection-checker-685895c7c9-mhsj5\" (UID: \"159cde9b-22a5-4930-9bf6-2535ab4af3d4\") " pod="kube-system/kas-connection-checker-685895c7c9-mhsj5" Oct 30 16:47:03.653099 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.653082 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nttq\" (UniqueName: \"kubernetes.io/projected/8c30c30a-8c1f-4552-b2d1-2692bc423d45-kube-api-access-9nttq\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.656322 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.656303 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q52w4\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-kube-api-access-q52w4\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.660243 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.660219 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqff6\" (UniqueName: \"kubernetes.io/projected/f13fd623-29b9-4d8b-b76d-668ca123010f-kube-api-access-sqff6\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:03.660350 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.660291 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwhw\" (UniqueName: \"kubernetes.io/projected/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-kube-api-access-rvwhw\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:47:03.660386 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.660360 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvsrx\" (UniqueName: \"kubernetes.io/projected/a41c35af-f159-4802-9b53-fbb8cbee2104-kube-api-access-rvsrx\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.664634 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.661980 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2qd\" (UniqueName: \"kubernetes.io/projected/b174c80c-af6a-4f17-8442-1a029b286011-kube-api-access-6w2qd\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:47:03.665800 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.665782 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-bound-sa-token\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.667458 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.667442 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xjxv\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-kube-api-access-7xjxv\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.680964 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.680946 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svsvb\" (UniqueName: \"kubernetes.io/projected/3ca4c236-e28d-4331-9336-63a71d706ef0-kube-api-access-svsvb\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:47:03.681999 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.681984 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp752\" (UniqueName: \"kubernetes.io/projected/852f74bb-c91e-4075-8335-aeba3e723709-kube-api-access-vp752\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.684863 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.684841 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwc77\" (UniqueName: \"kubernetes.io/projected/883b14e2-b8bc-4650-a638-c33673683a45-kube-api-access-fwc77\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:03.685323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.685305 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72lmm\" (UniqueName: \"kubernetes.io/projected/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-kube-api-access-72lmm\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:47:03.686802 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.686782 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-774sr\" (UniqueName: \"kubernetes.io/projected/252c7435-f4c1-4dac-a468-e3287c6a7e9e-kube-api-access-774sr\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.687703 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.687640 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-bound-sa-token\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.688022 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.688002 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzz5\" (UniqueName: \"kubernetes.io/projected/a132686f-9475-437d-826e-22fb93e86323-kube-api-access-cmzz5\") pod \"network-check-target-scdcl\" (UID: \"a132686f-9475-437d-826e-22fb93e86323\") " pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:47:03.688211 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.688187 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cmkh\" (UniqueName: \"kubernetes.io/projected/dacad254-b0c1-48fd-8377-1067d65ea230-kube-api-access-8cmkh\") pod \"console-operator-7cd56b8dbc-n758f\" (UID: \"dacad254-b0c1-48fd-8377-1067d65ea230\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:03.688379 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.688365 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrlc\" (UniqueName: \"kubernetes.io/projected/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-kube-api-access-pjrlc\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:03.689696 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.689663 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmgvx\" (UniqueName: \"kubernetes.io/projected/b34094c9-7349-4284-8b86-2253f9e4d87f-kube-api-access-zmgvx\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:47:03.692817 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.692799 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qprsq\" (UniqueName: \"kubernetes.io/projected/68321123-f8fc-4c6a-bd22-1f614a8a3b55-kube-api-access-qprsq\") pod \"ingress-canary-tb2dr\" (UID: \"68321123-f8fc-4c6a-bd22-1f614a8a3b55\") " pod="openshift-ingress-canary/ingress-canary-tb2dr" Oct 30 16:47:03.712808 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.712767 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-sys\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.712808 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.712816 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-slash\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.713013 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.712834 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b174c80c-af6a-4f17-8442-1a029b286011-host-slash\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:47:03.713013 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.712849 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/883b14e2-b8bc-4650-a638-c33673683a45-crio-socket\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:03.713013 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.712869 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:03.713013 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.712922 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-sys\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.713013 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.712943 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b174c80c-af6a-4f17-8442-1a029b286011-host-slash\") pod \"iptables-alerter-l4rsf\" (UID: \"b174c80c-af6a-4f17-8442-1a029b286011\") " pod="openshift-network-operator/iptables-alerter-l4rsf" Oct 30 16:47:03.713013 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.712956 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/883b14e2-b8bc-4650-a638-c33673683a45-crio-socket\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:03.713013 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.712969 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-host\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.713013 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.712922 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-slash\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.713013 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.712986 4828 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:47:03.713013 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.712998 4828 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67785c878-8c9pr: secret "image-registry-tls" not found Oct 30 16:47:03.713013 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.712996 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-run-ovn-kubernetes\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713024 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-kubelet-config\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713035 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-host\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713047 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-systemd\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.713061 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls podName:3794a50e-6c2f-40bc-9be7-eda1388f3178 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:04.213046497 +0000 UTC m=+1.858556655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls") pod "image-registry-67785c878-8c9pr" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178") : secret "image-registry-tls" not found Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713068 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-run-ovn-kubernetes\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713085 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-systemd\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713099 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713119 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713140 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-registration-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713158 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713159 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-system-cni-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.713178 4828 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713188 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-conf-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713206 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysctl-d\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713221 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-run\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713223 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-system-cni-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.713247 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:04.213232104 +0000 UTC m=+1.858742280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : secret "router-metrics-certs-default" not found Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713115 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-kubelet-config\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713204 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-registration-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713267 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713236 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-conf-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713274 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-run\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713296 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-var-lib-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.713321 4828 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.713334 4828 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d7cb6cf-rzmrh: secret "image-registry-tls" not found Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713351 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-etc-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713366 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysctl-d\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.713384 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls podName:d38312b4-bd6e-4a5f-8a33-d93fb827debc nodeName:}" failed. No retries permitted until 2026-10-30 16:47:04.213371235 +0000 UTC m=+1.858881394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls") pod "image-registry-797d7cb6cf-rzmrh" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc") : secret "image-registry-tls" not found Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713320 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-var-lib-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713398 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-etc-openvswitch\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713400 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-lib-modules\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713483 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-lib-modules\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713497 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-log-socket\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713513 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-cni-bin\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713539 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-system-cni-dir\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713554 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-kubelet\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713570 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-multus-certs\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713585 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-cni-bin\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713581 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-system-cni-dir\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713618 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-kubelet\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713617 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713610 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-log-socket\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713638 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-multus-certs\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713653 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-run-netns\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713657 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713678 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713709 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-systemd-units\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713720 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-run-netns\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.713738 4828 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713745 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-systemd-units\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713726 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-etc-kubernetes\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.713769 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert podName:7918c4b3-b4f4-4435-9de9-77fbe4906a7f nodeName:}" failed. No retries permitted until 2026-10-30 16:47:04.213757506 +0000 UTC m=+1.859267664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7zrs4" (UID: "7918c4b3-b4f4-4435-9de9-77fbe4906a7f") : secret "networking-console-plugin-cert" not found Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713772 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-etc-kubernetes\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713798 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-cni-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713813 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysctl-conf\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713828 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-netns\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713846 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-cni-bin\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713860 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-cni-dir\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713882 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-socket-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713890 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-netns\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713927 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-kubelet\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713937 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-cni-bin\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713950 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-node-log\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713964 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-kubelet\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713972 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysctl-conf\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713975 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-socket-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713988 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-node-log\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.713975 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-socket-dir-parent\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714019 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-k8s-cni-cncf-io\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714040 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-cni-multus\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714056 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-dbus\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714021 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-multus-socket-dir-parent\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714085 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714101 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-var-lib-cni-multus\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714108 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac8ed89d-9808-4bff-9d24-3e5fc34041ad-dbus\") pod \"global-pull-secret-syncer-wgxv6\" (UID: \"ac8ed89d-9808-4bff-9d24-3e5fc34041ad\") " pod="kube-system/global-pull-secret-syncer-wgxv6" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714110 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-host-run-k8s-cni-cncf-io\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714112 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-modprobe-d\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.714163 4828 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714161 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.714195 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:04.214182745 +0000 UTC m=+1.859692905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : secret "insights-runtime-extractor-tls" not found Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714206 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-cnibin\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714192 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-modprobe-d\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714221 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-hostroot\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714213 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714235 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-var-lib-kubelet\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714240 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-cnibin\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714262 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714266 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b54c34b7-e8c3-4fe9-9b66-36336dfae405-hostroot\") pod \"multus-q9f8r\" (UID: \"b54c34b7-e8c3-4fe9-9b66-36336dfae405\") " pod="openshift-multus/multus-q9f8r" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714277 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-cnibin\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714295 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714314 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-var-lib-kubelet\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714314 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-cni-netd\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714346 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-cnibin\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714358 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714371 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-host-cni-netd\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714386 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-kubernetes\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.714398 4828 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.714404 4828 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714412 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.714423 4828 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.714451 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs podName:3ca4c236-e28d-4331-9336-63a71d706ef0 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:04.214437391 +0000 UTC m=+1.859947552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs") pod "network-metrics-daemon-8mpgm" (UID: "3ca4c236-e28d-4331-9336-63a71d706ef0") : secret "metrics-daemon-secret" not found Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714463 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-kubernetes\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.714468 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls podName:dbb5e2c6-44f4-4152-a7ac-98ae4c045190 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:04.214458971 +0000 UTC m=+1.859969129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c69x2" (UID: "dbb5e2c6-44f4-4152-a7ac-98ae4c045190") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.714491 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:04.214481651 +0000 UTC m=+1.859991813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : configmap "kube-rbac-proxy" not found Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.714502 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:04.214496828 +0000 UTC m=+1.860006987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : configmap references non-existent config key: service-ca.crt Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714517 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-systemd\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714537 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-ovn\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714555 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-hosts-file\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714575 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714601 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-ovn\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714604 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-host\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714622 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8aeb454-5336-4fc1-8194-54d54a6cdb6c-hosts-file\") pod \"node-resolver-nbbqf\" (UID: \"f8aeb454-5336-4fc1-8194-54d54a6cdb6c\") " pod="openshift-dns/node-resolver-nbbqf" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714624 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/852f74bb-c91e-4075-8335-aeba3e723709-run-systemd\") pod \"ovnkube-node-2ljft\" (UID: \"852f74bb-c91e-4075-8335-aeba3e723709\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714643 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f-host\") pod \"node-ca-hfsmw\" (UID: \"e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f\") " pod="openshift-image-registry/node-ca-hfsmw" Oct 30 16:47:03.714625 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.714661 4828 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.714689 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls podName:b34094c9-7349-4284-8b86-2253f9e4d87f nodeName:}" failed. No retries permitted until 2026-10-30 16:47:04.214681519 +0000 UTC m=+1.860191698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fpc52" (UID: "b34094c9-7349-4284-8b86-2253f9e4d87f") : secret "samples-operator-tls" not found Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714701 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714717 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714732 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714746 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-device-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714761 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-sys-fs\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714778 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysconfig\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714789 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714807 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-device-dir\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.714836 4828 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714847 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a41c35af-f159-4802-9b53-fbb8cbee2104-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2hh98\" (UID: \"a41c35af-f159-4802-9b53-fbb8cbee2104\") " pod="openshift-multus/multus-additional-cni-plugins-2hh98" Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714846 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/252c7435-f4c1-4dac-a468-e3287c6a7e9e-sys-fs\") pod \"aws-ebs-csi-driver-node-sjmjl\" (UID: \"252c7435-f4c1-4dac-a468-e3287c6a7e9e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjmjl" Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:03.714872 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls podName:f13fd623-29b9-4d8b-b76d-668ca123010f nodeName:}" failed. No retries permitted until 2026-10-30 16:47:04.214859478 +0000 UTC m=+1.860369651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls") pod "dns-default-6rrfj" (UID: "f13fd623-29b9-4d8b-b76d-668ca123010f") : secret "dns-default-metrics-tls" not found Oct 30 16:47:03.716757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.714881 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c30c30a-8c1f-4552-b2d1-2692bc423d45-etc-sysconfig\") pod \"tuned-c7ckl\" (UID: \"8c30c30a-8c1f-4552-b2d1-2692bc423d45\") " pod="openshift-cluster-node-tuning-operator/tuned-c7ckl" Oct 30 16:47:03.817552 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:03.817519 4828 scope.go:122] "RemoveContainer" containerID="c99e50eea3562218c0bb5c17d257f8fbe287413615ed42c5c59b5e6a7a33f71f" Oct 30 16:47:04.219167 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.219079 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:04.219167 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.219125 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:47:04.219167 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.219149 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:47:04.219397 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.219173 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:04.219397 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.219199 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:04.219397 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.219220 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:47:04.219397 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219234 4828 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:47:04.219397 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219257 4828 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:47:04.219397 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.219285 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:04.219397 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219290 4828 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:47:04.219397 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219302 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.219287159 +0000 UTC m=+2.864797344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : secret "insights-runtime-extractor-tls" not found Oct 30 16:47:04.219397 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219241 4828 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:47:04.219397 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219342 4828 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:47:04.219397 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219353 4828 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:47:04.219397 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219343 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs podName:3ca4c236-e28d-4331-9336-63a71d706ef0 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.2193283 +0000 UTC m=+2.864838474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs") pod "network-metrics-daemon-8mpgm" (UID: "3ca4c236-e28d-4331-9336-63a71d706ef0") : secret "metrics-daemon-secret" not found Oct 30 16:47:04.219397 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219394 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.219381457 +0000 UTC m=+2.864891616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : configmap "kube-rbac-proxy" not found Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219411 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls podName:dbb5e2c6-44f4-4152-a7ac-98ae4c045190 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.219404838 +0000 UTC m=+2.864914996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c69x2" (UID: "dbb5e2c6-44f4-4152-a7ac-98ae4c045190") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219424 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.219417448 +0000 UTC m=+2.864927606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : configmap references non-existent config key: service-ca.crt Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219442 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls podName:b34094c9-7349-4284-8b86-2253f9e4d87f nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.219433037 +0000 UTC m=+2.864943196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-fpc52" (UID: "b34094c9-7349-4284-8b86-2253f9e4d87f") : secret "samples-operator-tls" not found Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219457 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls podName:f13fd623-29b9-4d8b-b76d-668ca123010f nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.219451253 +0000 UTC m=+2.864961411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls") pod "dns-default-6rrfj" (UID: "f13fd623-29b9-4d8b-b76d-668ca123010f") : secret "dns-default-metrics-tls" not found Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.219480 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.219498 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.219521 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219540 4828 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.219555 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219564 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.219557676 +0000 UTC m=+2.865067839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : secret "router-metrics-certs-default" not found Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219600 4828 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219619 4828 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67785c878-8c9pr: secret "image-registry-tls" not found Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219606 4828 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219647 4828 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219656 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert podName:7918c4b3-b4f4-4435-9de9-77fbe4906a7f nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.219646714 +0000 UTC m=+2.865156887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-7zrs4" (UID: "7918c4b3-b4f4-4435-9de9-77fbe4906a7f") : secret "networking-console-plugin-cert" not found Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219659 4828 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-797d7cb6cf-rzmrh: secret "image-registry-tls" not found Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219675 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls podName:3794a50e-6c2f-40bc-9be7-eda1388f3178 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.219665382 +0000 UTC m=+2.865175541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls") pod "image-registry-67785c878-8c9pr" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178") : secret "image-registry-tls" not found Oct 30 16:47:04.219868 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.219692 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls podName:d38312b4-bd6e-4a5f-8a33-d93fb827debc nodeName:}" failed. No retries permitted until 2026-10-30 16:47:05.219682045 +0000 UTC m=+2.865192222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls") pod "image-registry-797d7cb6cf-rzmrh" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc") : secret "image-registry-tls" not found Oct 30 16:47:04.455202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.455163 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:04.538496 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.538469 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 16:47:04.538879 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.538862 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/1.log" Oct 30 16:47:04.539008 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.538917 4828 generic.go:356] "Generic (PLEG): container finished" podID="dacad254-b0c1-48fd-8377-1067d65ea230" containerID="b1757bc9e3a7b9e9abc40cc6faf28bdf2cba67cc4ab46e4795ef994e7cca0f29" exitCode=255 Oct 30 16:47:04.539008 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.538956 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" event={"ID":"dacad254-b0c1-48fd-8377-1067d65ea230","Type":"ContainerDied","Data":"b1757bc9e3a7b9e9abc40cc6faf28bdf2cba67cc4ab46e4795ef994e7cca0f29"} Oct 30 16:47:04.539098 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.539008 4828 scope.go:122] "RemoveContainer" containerID="c99e50eea3562218c0bb5c17d257f8fbe287413615ed42c5c59b5e6a7a33f71f" Oct 30 16:47:04.539406 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.539385 4828 scope.go:122] "RemoveContainer" containerID="b1757bc9e3a7b9e9abc40cc6faf28bdf2cba67cc4ab46e4795ef994e7cca0f29" Oct 30 16:47:04.539634 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:04.539604 4828 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n758f_openshift-console-operator(dacad254-b0c1-48fd-8377-1067d65ea230)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" podUID="dacad254-b0c1-48fd-8377-1067d65ea230" Oct 30 16:47:04.614496 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.614459 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:04.821073 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:04.820985 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:05.139077 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.138993 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:05.230641 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.230600 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:05.230641 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.230645 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.230668 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.230691 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.230725 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.230763 4828 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.230773 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.230816 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.230828 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.230811893 +0000 UTC m=+4.876322052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : secret "router-metrics-certs-default" not found Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.230763 4828 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.230876 4828 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.230884 4828 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.230928 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls podName:f13fd623-29b9-4d8b-b76d-668ca123010f nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.230894955 +0000 UTC m=+4.876405127 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls") pod "dns-default-6rrfj" (UID: "f13fd623-29b9-4d8b-b76d-668ca123010f") : secret "dns-default-metrics-tls" not found Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.230887 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.230954 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls podName:dbb5e2c6-44f4-4152-a7ac-98ae4c045190 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.230936365 +0000 UTC m=+4.876446540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c69x2" (UID: "dbb5e2c6-44f4-4152-a7ac-98ae4c045190") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.230959 4828 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.230972 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.230963791 +0000 UTC m=+4.876473950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : secret "insights-runtime-extractor-tls" not found Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.230994 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs podName:3ca4c236-e28d-4331-9336-63a71d706ef0 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.230984177 +0000 UTC m=+4.876494334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs") pod "network-metrics-daemon-8mpgm" (UID: "3ca4c236-e28d-4331-9336-63a71d706ef0") : secret "metrics-daemon-secret" not found Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.231012 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:05.231043 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.231042 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:05.231815 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.231042 4828 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:47:05.231815 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.231080 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:47:05.231815 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.231123 4828 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:47:05.231815 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.231151 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.231141402 +0000 UTC m=+4.876651565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : configmap references non-existent config key: service-ca.crt Oct 30 16:47:05.231815 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.231187 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:07.231170998 +0000 UTC m=+4.876681171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : configmap "kube-rbac-proxy" not found Oct 30 16:47:05.233327 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.233303 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b34094c9-7349-4284-8b86-2253f9e4d87f-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-fpc52\" (UID: \"b34094c9-7349-4284-8b86-2253f9e4d87f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:47:05.233423 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.233370 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\") pod \"image-registry-67785c878-8c9pr\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:05.233470 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.233436 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7918c4b3-b4f4-4435-9de9-77fbe4906a7f-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-7zrs4\" (UID: \"7918c4b3-b4f4-4435-9de9-77fbe4906a7f\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:47:05.233507 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.233471 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\") pod \"image-registry-797d7cb6cf-rzmrh\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:05.295932 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.295881 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:05.316276 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.316245 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:05.318370 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.318348 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" Oct 30 16:47:05.319829 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.319808 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" Oct 30 16:47:05.325793 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.325735 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:05.544171 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.544139 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 16:47:05.630121 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.630099 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-7zrs4"] Oct 30 16:47:05.633880 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:05.632514 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7918c4b3_b4f4_4435_9de9_77fbe4906a7f.slice/crio-e88336ec00903d22a7200604d1667ec61e446b54c8ee5a55d24111c3c95eedc0 WatchSource:0}: Error finding container e88336ec00903d22a7200604d1667ec61e446b54c8ee5a55d24111c3c95eedc0: Status 404 returned error can't find the container with id e88336ec00903d22a7200604d1667ec61e446b54c8ee5a55d24111c3c95eedc0 Oct 30 16:47:05.636784 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.636766 4828 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:47:05.654035 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.654015 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52"] Oct 30 16:47:05.660595 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.660571 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67785c878-8c9pr"] Oct 30 16:47:05.666241 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.666214 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-797d7cb6cf-rzmrh"] Oct 30 16:47:05.891644 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.891598 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:05.998969 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.998873 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:05.999358 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:05.999337 4828 scope.go:122] "RemoveContainer" containerID="b1757bc9e3a7b9e9abc40cc6faf28bdf2cba67cc4ab46e4795ef994e7cca0f29" Oct 30 16:47:05.999598 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:05.999572 4828 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n758f_openshift-console-operator(dacad254-b0c1-48fd-8377-1067d65ea230)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" podUID="dacad254-b0c1-48fd-8377-1067d65ea230" Oct 30 16:47:06.061036 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:06.060957 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:06.318879 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:06.318841 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:06.514972 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:06.514935 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:06.549672 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:06.549593 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" event={"ID":"b34094c9-7349-4284-8b86-2253f9e4d87f","Type":"ContainerStarted","Data":"ed3fdf477d79ac5361e06e63c8fd2f38fbbfdf15e91be8bafd254ace6b5b46f6"} Oct 30 16:47:06.551541 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:06.551516 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67785c878-8c9pr" event={"ID":"3794a50e-6c2f-40bc-9be7-eda1388f3178","Type":"ContainerStarted","Data":"e8ce4cccfe26d642777b11e968215b78f68040dbe9e56584f30858862a849a72"} Oct 30 16:47:06.551677 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:06.551559 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67785c878-8c9pr" event={"ID":"3794a50e-6c2f-40bc-9be7-eda1388f3178","Type":"ContainerStarted","Data":"881ae6a0e628e80ae92e2e943cf669a586842320ace0ac3ba5d83f7f90d307b5"} Oct 30 16:47:06.551677 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:06.551614 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:06.553407 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:06.553379 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" event={"ID":"d38312b4-bd6e-4a5f-8a33-d93fb827debc","Type":"ContainerStarted","Data":"7a382cc70e2ec4154df1cbb4f510025c8c98b806b6b3e0b963d3136b40329edf"} Oct 30 16:47:06.553556 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:06.553425 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" event={"ID":"d38312b4-bd6e-4a5f-8a33-d93fb827debc","Type":"ContainerStarted","Data":"ae151e4f05975668309df9c14f8bcfca724c4682c2bb60814b7a8fde4905800a"} Oct 30 16:47:06.556012 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:06.555987 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" event={"ID":"7918c4b3-b4f4-4435-9de9-77fbe4906a7f","Type":"ContainerStarted","Data":"e88336ec00903d22a7200604d1667ec61e446b54c8ee5a55d24111c3c95eedc0"} Oct 30 16:47:06.712966 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:06.712926 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:06.814739 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:06.814710 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nbbqf_f8aeb454-5336-4fc1-8194-54d54a6cdb6c/dns-node-resolver/0.log" Oct 30 16:47:07.133175 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.133130 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:07.211833 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.211809 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67785c878-8c9pr_3794a50e-6c2f-40bc-9be7-eda1388f3178/registry/0.log" Oct 30 16:47:07.250405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.250381 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:07.250522 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.250418 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:47:07.250522 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.250438 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:47:07.250522 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.250454 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:07.250522 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.250485 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:07.250522 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.250521 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:07.250777 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:07.250526 4828 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:47:07.250777 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:07.250533 4828 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:47:07.250777 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:07.250557 4828 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:47:07.250777 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:07.250526 4828 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:47:07.250777 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:07.250576 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls podName:dbb5e2c6-44f4-4152-a7ac-98ae4c045190 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:11.250562605 +0000 UTC m=+8.896072763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-c69x2" (UID: "dbb5e2c6-44f4-4152-a7ac-98ae4c045190") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:47:07.250777 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:07.250600 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:11.250583593 +0000 UTC m=+8.896093751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : configmap "kube-rbac-proxy" not found Oct 30 16:47:07.250777 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:07.250614 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs podName:3ca4c236-e28d-4331-9336-63a71d706ef0 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:11.250607911 +0000 UTC m=+8.896118070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs") pod "network-metrics-daemon-8mpgm" (UID: "3ca4c236-e28d-4331-9336-63a71d706ef0") : secret "metrics-daemon-secret" not found Oct 30 16:47:07.250777 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:07.250623 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:11.250618497 +0000 UTC m=+8.896128654 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : secret "insights-runtime-extractor-tls" not found Oct 30 16:47:07.250777 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.250557 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:07.250777 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:07.250635 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle podName:a6361ea7-1d84-4bc8-b439-3f60ed9d9e40 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:11.25062867 +0000 UTC m=+8.896138828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle") pod "router-default-5c78887cdf-9jhzv" (UID: "a6361ea7-1d84-4bc8-b439-3f60ed9d9e40") : configmap references non-existent config key: service-ca.crt Oct 30 16:47:07.250777 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:07.250646 4828 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:47:07.250777 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:07.250684 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls podName:f13fd623-29b9-4d8b-b76d-668ca123010f nodeName:}" failed. No retries permitted until 2026-10-30 16:47:11.250671026 +0000 UTC m=+8.896181205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls") pod "dns-default-6rrfj" (UID: "f13fd623-29b9-4d8b-b76d-668ca123010f") : secret "dns-default-metrics-tls" not found Oct 30 16:47:07.252574 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.252539 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-metrics-certs\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:07.409809 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.409726 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-797d7cb6cf-rzmrh_d38312b4-bd6e-4a5f-8a33-d93fb827debc/registry/0.log" Oct 30 16:47:07.561039 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.561000 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-7zrs4" event={"ID":"7918c4b3-b4f4-4435-9de9-77fbe4906a7f","Type":"ContainerStarted","Data":"72ee5bd78882f22ae8a264f6cc1a3a758ae57dfeedcdb403c59e38212a780bb4"} Oct 30 16:47:07.562800 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.562768 4828 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:47:07.562800 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.562781 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" event={"ID":"b34094c9-7349-4284-8b86-2253f9e4d87f","Type":"ContainerStarted","Data":"0128dc29d63c8fbf69aa5c2683bbd8724e63fca60b3348fa674fc5cf2fadd872"} Oct 30 16:47:07.562963 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.562829 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-fpc52" event={"ID":"b34094c9-7349-4284-8b86-2253f9e4d87f","Type":"ContainerStarted","Data":"8269f1cb842c965d6ccb629b7ec422b616fab7d253f21b4b2dbec81876724edc"} Oct 30 16:47:07.712776 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.712700 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:07.807751 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:07.807723 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hfsmw_e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f/node-ca/0.log" Oct 30 16:47:08.109259 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:08.109221 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:08.514494 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:08.514461 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:08.710713 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:08.710666 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:08.809446 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:08.809370 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tb2dr_68321123-f8fc-4c6a-bd22-1f614a8a3b55/serve-healthcheck-canary/0.log" Oct 30 16:47:09.322847 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:09.322809 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:09.409345 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:09.409313 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq_35a14012-73eb-4f3b-b762-f56c6a963e44/kube-storage-version-migrator-operator/0.log" Oct 30 16:47:09.793986 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:09.793939 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:09.903123 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:09.903086 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:10.048513 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:10.048420 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:10.200403 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:10.200364 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:10.507784 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:10.507701 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:10.717257 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:10.717226 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:10.768378 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:10.768348 4828 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:10.768772 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:10.768758 4828 scope.go:122] "RemoveContainer" containerID="b1757bc9e3a7b9e9abc40cc6faf28bdf2cba67cc4ab46e4795ef994e7cca0f29" Oct 30 16:47:10.768980 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:10.768962 4828 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-n758f_openshift-console-operator(dacad254-b0c1-48fd-8377-1067d65ea230)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" podUID="dacad254-b0c1-48fd-8377-1067d65ea230" Oct 30 16:47:10.769819 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:10.769801 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:47:11.289300 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.289272 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:11.289746 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.289344 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:11.289746 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.289371 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:47:11.289746 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.289389 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:47:11.289746 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.289405 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:11.289746 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.289422 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:11.289746 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:11.289420 4828 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:47:11.289746 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:11.289475 4828 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:47:11.289746 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:11.289494 4828 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:47:11.289746 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:11.289512 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls podName:f13fd623-29b9-4d8b-b76d-668ca123010f nodeName:}" failed. No retries permitted until 2026-10-30 16:47:19.289495699 +0000 UTC m=+16.935005857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls") pod "dns-default-6rrfj" (UID: "f13fd623-29b9-4d8b-b76d-668ca123010f") : secret "dns-default-metrics-tls" not found Oct 30 16:47:11.289746 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:11.289534 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs podName:3ca4c236-e28d-4331-9336-63a71d706ef0 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:19.289520275 +0000 UTC m=+16.935030452 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs") pod "network-metrics-daemon-8mpgm" (UID: "3ca4c236-e28d-4331-9336-63a71d706ef0") : secret "metrics-daemon-secret" not found Oct 30 16:47:11.289746 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:11.289482 4828 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:47:11.289746 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:11.289549 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:19.289543532 +0000 UTC m=+16.935053689 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : secret "insights-runtime-extractor-tls" not found Oct 30 16:47:11.289746 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:11.289594 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm podName:883b14e2-b8bc-4650-a638-c33673683a45 nodeName:}" failed. No retries permitted until 2026-10-30 16:47:19.289577926 +0000 UTC m=+16.935088085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7sxm" (UID: "883b14e2-b8bc-4650-a638-c33673683a45") : configmap "kube-rbac-proxy" not found Oct 30 16:47:11.290443 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.290415 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6361ea7-1d84-4bc8-b439-3f60ed9d9e40-service-ca-bundle\") pod \"router-default-5c78887cdf-9jhzv\" (UID: \"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40\") " pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:11.291793 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.291773 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dbb5e2c6-44f4-4152-a7ac-98ae4c045190-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-c69x2\" (UID: \"dbb5e2c6-44f4-4152-a7ac-98ae4c045190\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:47:11.315762 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.315732 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:11.317849 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.317829 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" Oct 30 16:47:11.320689 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.320659 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:11.479883 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.479855 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:11.623286 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.623258 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c78887cdf-9jhzv"] Oct 30 16:47:11.624057 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:11.624032 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6361ea7_1d84_4bc8_b439_3f60ed9d9e40.slice/crio-2273056b55c4429670a01098083872b8e3ef31d2ecb89d7b07e877abd69d7e9e WatchSource:0}: Error finding container 2273056b55c4429670a01098083872b8e3ef31d2ecb89d7b07e877abd69d7e9e: Status 404 returned error can't find the container with id 2273056b55c4429670a01098083872b8e3ef31d2ecb89d7b07e877abd69d7e9e Oct 30 16:47:11.631665 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.631643 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2"] Oct 30 16:47:11.632955 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:11.632933 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb5e2c6_44f4_4152_a7ac_98ae4c045190.slice/crio-6c0c6222b594e480b9e02f37e77c2c3f160d8ee762bf49b1e96eba4dc903c8cd WatchSource:0}: Error finding container 6c0c6222b594e480b9e02f37e77c2c3f160d8ee762bf49b1e96eba4dc903c8cd: Status 404 returned error can't find the container with id 6c0c6222b594e480b9e02f37e77c2c3f160d8ee762bf49b1e96eba4dc903c8cd Oct 30 16:47:11.918249 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:11.918171 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:12.108722 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:12.108686 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:12.473408 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:12.473367 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:12.581659 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:12.581437 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c78887cdf-9jhzv" event={"ID":"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40","Type":"ContainerStarted","Data":"0aff15bf8055134800380028055142ce9f27f47519d43d4c5bd028e98d77e572"} Oct 30 16:47:12.581659 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:12.581492 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c78887cdf-9jhzv" event={"ID":"a6361ea7-1d84-4bc8-b439-3f60ed9d9e40","Type":"ContainerStarted","Data":"2273056b55c4429670a01098083872b8e3ef31d2ecb89d7b07e877abd69d7e9e"} Oct 30 16:47:12.582823 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:12.582798 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" event={"ID":"dbb5e2c6-44f4-4152-a7ac-98ae4c045190","Type":"ContainerStarted","Data":"6c0c6222b594e480b9e02f37e77c2c3f160d8ee762bf49b1e96eba4dc903c8cd"} Oct 30 16:47:12.641344 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:12.641306 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:12.794934 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:12.794876 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:12.945571 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:12.945535 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:13.309126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:13.309087 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:13.315944 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:13.315916 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:13.318758 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:13.318737 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:13.588241 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:13.588209 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-c69x2" event={"ID":"dbb5e2c6-44f4-4152-a7ac-98ae4c045190","Type":"ContainerStarted","Data":"62c133c6ddb40856104c68e92c0afdf2f3a222b6e1189919d3250dda19439c91"} Oct 30 16:47:13.655352 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:13.655322 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:13.655544 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:13.655516 4828 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:47:13.846286 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:13.846211 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:13.999745 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:13.999711 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:14.152592 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:14.152492 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:14.334999 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:14.334959 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:14.483144 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:14.483069 4828 scope.go:122] "RemoveContainer" containerID="b1757bc9e3a7b9e9abc40cc6faf28bdf2cba67cc4ab46e4795ef994e7cca0f29" Oct 30 16:47:14.504991 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:14.504963 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:14.593107 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:14.593078 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 16:47:14.593447 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:14.593190 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" event={"ID":"dacad254-b0c1-48fd-8377-1067d65ea230","Type":"ContainerStarted","Data":"8e8ed4f54d0cbf4c16e20a4ad5ace18f60eabcc2c7c8819e8131ca4dbe3ea0a9"} Oct 30 16:47:14.593447 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:14.593216 4828 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:47:14.593725 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:14.593697 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:14.594781 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:14.594758 4828 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-n758f container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" start-of-body= Oct 30 16:47:14.594876 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:14.594821 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" podUID="dacad254-b0c1-48fd-8377-1067d65ea230" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" Oct 30 16:47:14.705538 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:14.705501 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:14.911294 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:14.911261 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:15.084769 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:15.084730 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:15.190070 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:15.189982 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:15.191401 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:15.191377 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5c78887cdf-9jhzv" Oct 30 16:47:15.243385 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:15.243352 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:15.394630 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:15.394592 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:15.601169 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:15.601144 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-n758f" Oct 30 16:47:15.677169 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:15.677135 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:15.827523 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:15.827487 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:16.482924 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:16.482865 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:16.502281 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:16.502247 4828 patch_prober.go:29] interesting pod/image-registry-797d7cb6cf-rzmrh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 16:47:16.502429 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:16.502293 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" podUID="d38312b4-bd6e-4a5f-8a33-d93fb827debc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:47:16.634170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:16.634139 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:19.369647 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.369605 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:19.370207 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.369702 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:19.370207 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.369751 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:47:19.370207 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.369809 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:19.370344 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.370323 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/883b14e2-b8bc-4650-a638-c33673683a45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:19.371957 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.371937 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f13fd623-29b9-4d8b-b76d-668ca123010f-metrics-tls\") pod \"dns-default-6rrfj\" (UID: \"f13fd623-29b9-4d8b-b76d-668ca123010f\") " pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:19.372062 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.372021 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/883b14e2-b8bc-4650-a638-c33673683a45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7sxm\" (UID: \"883b14e2-b8bc-4650-a638-c33673683a45\") " pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:19.372174 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.372156 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ca4c236-e28d-4331-9336-63a71d706ef0-metrics-certs\") pod \"network-metrics-daemon-8mpgm\" (UID: \"3ca4c236-e28d-4331-9336-63a71d706ef0\") " pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:47:19.421792 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.421754 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mpgm" Oct 30 16:47:19.426478 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.426456 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:19.426569 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.426550 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7sxm" Oct 30 16:47:19.723226 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.723166 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8mpgm"] Oct 30 16:47:19.725521 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:19.725493 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca4c236_e28d_4331_9336_63a71d706ef0.slice/crio-538cabb1ba7bd63fccb0bb38b3332df50845f519535b83ee8d2c59bd2d7e57aa WatchSource:0}: Error finding container 538cabb1ba7bd63fccb0bb38b3332df50845f519535b83ee8d2c59bd2d7e57aa: Status 404 returned error can't find the container with id 538cabb1ba7bd63fccb0bb38b3332df50845f519535b83ee8d2c59bd2d7e57aa Oct 30 16:47:19.762650 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.762627 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7sxm"] Oct 30 16:47:19.764662 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:19.764637 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883b14e2_b8bc_4650_a638_c33673683a45.slice/crio-a36fe48d6bb4553fcac2d2982232775b3912634b30ea8b77c5d73fcba931cd82 WatchSource:0}: Error finding container a36fe48d6bb4553fcac2d2982232775b3912634b30ea8b77c5d73fcba931cd82: Status 404 returned error can't find the container with id a36fe48d6bb4553fcac2d2982232775b3912634b30ea8b77c5d73fcba931cd82 Oct 30 16:47:19.774087 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:19.774061 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13fd623_29b9_4d8b_b76d_668ca123010f.slice/crio-d3ef5a262b6d18a1c935dea4ba56597f854ed733e21cabeb01407655e8c77fc0 WatchSource:0}: Error finding container d3ef5a262b6d18a1c935dea4ba56597f854ed733e21cabeb01407655e8c77fc0: Status 404 returned error can't find the container with id d3ef5a262b6d18a1c935dea4ba56597f854ed733e21cabeb01407655e8c77fc0 Oct 30 16:47:19.776442 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:19.776414 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6rrfj"] Oct 30 16:47:20.539839 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:20.538436 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67785c878-8c9pr"] Oct 30 16:47:20.552997 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:20.550596 4828 patch_prober.go:29] interesting pod/image-registry-67785c878-8c9pr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 16:47:20.553161 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:20.553038 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-67785c878-8c9pr" podUID="3794a50e-6c2f-40bc-9be7-eda1388f3178" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:47:20.619544 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:20.619502 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8mpgm" event={"ID":"3ca4c236-e28d-4331-9336-63a71d706ef0","Type":"ContainerStarted","Data":"538cabb1ba7bd63fccb0bb38b3332df50845f519535b83ee8d2c59bd2d7e57aa"} Oct 30 16:47:20.622258 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:20.622220 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7sxm" event={"ID":"883b14e2-b8bc-4650-a638-c33673683a45","Type":"ContainerStarted","Data":"eb876e81819e5d0d329f09be0be91efc995125758e3156fd6cdef27dd0d26e10"} Oct 30 16:47:20.622397 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:20.622288 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7sxm" event={"ID":"883b14e2-b8bc-4650-a638-c33673683a45","Type":"ContainerStarted","Data":"c7ab5563c994904b1df9105c2e5ae96d4b589db7f21efc2c1fec1f1e765e0606"} Oct 30 16:47:20.622397 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:20.622312 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7sxm" event={"ID":"883b14e2-b8bc-4650-a638-c33673683a45","Type":"ContainerStarted","Data":"a36fe48d6bb4553fcac2d2982232775b3912634b30ea8b77c5d73fcba931cd82"} Oct 30 16:47:20.623996 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:20.623964 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6rrfj" event={"ID":"f13fd623-29b9-4d8b-b76d-668ca123010f","Type":"ContainerStarted","Data":"d3ef5a262b6d18a1c935dea4ba56597f854ed733e21cabeb01407655e8c77fc0"} Oct 30 16:47:20.783024 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:20.782987 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:20.937377 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:20.937350 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:21.631640 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:21.631601 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8mpgm" event={"ID":"3ca4c236-e28d-4331-9336-63a71d706ef0","Type":"ContainerStarted","Data":"7fcbbc9fa11f42f57eccf02b7ed0ebc9a5915cb39d7a911ef6fb227a467328cb"} Oct 30 16:47:21.632118 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:21.631704 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8mpgm" event={"ID":"3ca4c236-e28d-4331-9336-63a71d706ef0","Type":"ContainerStarted","Data":"12cba51562b29c16d4816228dc54910aba5003d56e9d76ad6b1d7a9cb750ff62"} Oct 30 16:47:22.637047 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:22.637008 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7sxm" event={"ID":"883b14e2-b8bc-4650-a638-c33673683a45","Type":"ContainerStarted","Data":"c9bcdb8acfd4c3f85e66a1c870585424cd5dd2d3deaeb1cff3efb4fcb7fc1b95"} Oct 30 16:47:22.638888 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:22.638864 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6rrfj" event={"ID":"f13fd623-29b9-4d8b-b76d-668ca123010f","Type":"ContainerStarted","Data":"de73078cb9d6801a053e227fcf5d2b87354cebb87931e874e0bc542e3a1c815e"} Oct 30 16:47:22.638962 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:22.638926 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6rrfj" event={"ID":"f13fd623-29b9-4d8b-b76d-668ca123010f","Type":"ContainerStarted","Data":"657463adacf4c31017d62d33a84fcb7dad4a6b56079e77f1189cc066e52a2839"} Oct 30 16:47:22.639015 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:22.638960 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:23.356152 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:23.356121 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:23.524657 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:23.524616 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:23.670150 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:23.670055 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:24.261109 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:24.261078 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ljft" Oct 30 16:47:26.500728 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:26.500691 4828 patch_prober.go:29] interesting pod/image-registry-797d7cb6cf-rzmrh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 16:47:26.501304 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:26.500744 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" podUID="d38312b4-bd6e-4a5f-8a33-d93fb827debc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:47:26.502326 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:26.502289 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:27.855387 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:27.855347 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:28.031891 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.031856 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:28.199209 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.199132 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dw2pc"] Oct 30 16:47:28.210073 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.210036 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:28.224856 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.224831 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.255766 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.255737 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 16:47:28.255958 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.255803 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 16:47:28.255958 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.255859 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 16:47:28.256076 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.255982 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 16:47:28.256076 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.256035 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rb7tp\"" Oct 30 16:47:28.340639 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.340603 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-root\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.340639 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.340641 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-textfile\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.340836 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.340678 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-tls\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.340836 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.340796 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-wtmp\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.340925 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.340838 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-metrics-client-ca\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.340925 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.340893 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prhzn\" (UniqueName: \"kubernetes.io/projected/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-kube-api-access-prhzn\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.340994 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.340960 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.341027 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.340995 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-sys\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.341081 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.341066 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.375498 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.375464 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:28.441893 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.441859 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prhzn\" (UniqueName: \"kubernetes.io/projected/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-kube-api-access-prhzn\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442074 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.441942 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442074 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.441974 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-sys\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442074 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.441999 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442074 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.442023 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-root\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442074 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.442040 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-textfile\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442374 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.442077 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-tls\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442374 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.442098 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-sys\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442374 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.442128 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-root\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442374 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.442114 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-wtmp\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442374 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.442223 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-metrics-client-ca\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442374 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.442243 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-wtmp\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442613 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.442469 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-textfile\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442613 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.442571 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-accelerators-collector-config\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.442693 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.442671 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-metrics-client-ca\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.444409 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.444386 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.444535 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.444493 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-node-exporter-tls\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.468488 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.468416 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prhzn\" (UniqueName: \"kubernetes.io/projected/3ffdad02-6e93-49ac-97d7-25f5a6ed61c4-kube-api-access-prhzn\") pod \"node-exporter-dw2pc\" (UID: \"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4\") " pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.559519 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.559467 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:28.574970 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.574939 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dw2pc" Oct 30 16:47:28.615872 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:47:28.615843 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffdad02_6e93_49ac_97d7_25f5a6ed61c4.slice/crio-0d8e4d90eb3357d2cda8c2c63277ca6f56d728cbf95a243c6b1a45f6a3709b79 WatchSource:0}: Error finding container 0d8e4d90eb3357d2cda8c2c63277ca6f56d728cbf95a243c6b1a45f6a3709b79: Status 404 returned error can't find the container with id 0d8e4d90eb3357d2cda8c2c63277ca6f56d728cbf95a243c6b1a45f6a3709b79 Oct 30 16:47:28.661154 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.661117 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dw2pc" event={"ID":"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4","Type":"ContainerStarted","Data":"0d8e4d90eb3357d2cda8c2c63277ca6f56d728cbf95a243c6b1a45f6a3709b79"} Oct 30 16:47:28.720824 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:28.720738 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:29.356118 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:29.356075 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:29.502470 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:29.502438 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:29.690727 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:29.690637 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:29.798370 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:29.798338 4828 patch_prober.go:29] interesting pod/image-registry-797d7cb6cf-rzmrh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 16:47:29.798532 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:29.798388 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" podUID="d38312b4-bd6e-4a5f-8a33-d93fb827debc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:47:30.178641 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:30.178603 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:30.380628 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:30.380584 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:30.544109 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:30.544077 4828 patch_prober.go:29] interesting pod/image-registry-67785c878-8c9pr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 16:47:30.544270 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:30.544125 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-67785c878-8c9pr" podUID="3794a50e-6c2f-40bc-9be7-eda1388f3178" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:47:30.672849 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:30.672764 4828 generic.go:356] "Generic (PLEG): container finished" podID="3ffdad02-6e93-49ac-97d7-25f5a6ed61c4" containerID="bb155c84a163ef134f052e8be9f4ad5d4d45a6f9fecaea2ebdd210588f87a99f" exitCode=0 Oct 30 16:47:30.673042 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:30.672851 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dw2pc" event={"ID":"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4","Type":"ContainerDied","Data":"bb155c84a163ef134f052e8be9f4ad5d4d45a6f9fecaea2ebdd210588f87a99f"} Oct 30 16:47:30.772886 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:30.772858 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-scdcl" Oct 30 16:47:31.423603 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:31.423564 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:31.627764 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:31.627735 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:31.679232 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:31.679150 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dw2pc" event={"ID":"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4","Type":"ContainerStarted","Data":"5b210a7e8beee15417c08928b6f2f4669a306729aed507a20e0f61e2622c6792"} Oct 30 16:47:31.679232 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:31.679200 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dw2pc" event={"ID":"3ffdad02-6e93-49ac-97d7-25f5a6ed61c4","Type":"ContainerStarted","Data":"23c235697e40002981fc16944c8c0909107753f6c098f54b63047bd074d6f317"} Oct 30 16:47:31.727733 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:31.727682 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dw2pc" podStartSLOduration=2.361343069 podStartE2EDuration="3.727668345s" totalImagesPullingTime="1.284771567s" totalInitContainerRuntime="81.553709ms" isStatefulPod=true podCreationTimestamp="2026-10-30 16:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:31.726976179 +0000 UTC m=+29.372486360" watchObservedRunningTime="2026-10-30 16:47:31.727668345 +0000 UTC m=+29.373178524" Oct 30 16:47:31.836726 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:31.836692 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:32.645067 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:32.645037 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6rrfj" Oct 30 16:47:32.778102 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:32.778070 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4hlmr" Oct 30 16:47:32.948718 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:32.948627 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:33.114404 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:33.114367 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:33.268853 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:33.268806 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:33.906494 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:33.906460 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:34.058816 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:34.058773 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:35.293633 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:35.293597 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:37.694328 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:37.694280 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:37.909060 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:47:37.909022 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:47:38.950134 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:38.950096 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:39.097449 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:39.097408 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:39.506443 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:39.506399 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:39.654693 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:39.654657 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:39.798665 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:39.798641 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:47:40.217595 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:40.217511 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:40.381454 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:40.381415 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:40.544137 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:40.544111 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:41.296552 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:41.296518 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:41.443051 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:41.443015 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:42.808186 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:42.808149 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:42.977918 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:42.977868 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:43.337530 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:43.337497 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-797d7cb6cf-rzmrh"] Oct 30 16:47:43.508985 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:43.508955 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:43.670998 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:43.670893 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:45.312879 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.312834 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:45.469700 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.469661 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:45.571053 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.570963 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67785c878-8c9pr" podUID="3794a50e-6c2f-40bc-9be7-eda1388f3178" containerName="registry" containerID="cri-o://e8ce4cccfe26d642777b11e968215b78f68040dbe9e56584f30858862a849a72" gracePeriod=30 Oct 30 16:47:45.731339 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.731305 4828 generic.go:356] "Generic (PLEG): container finished" podID="3794a50e-6c2f-40bc-9be7-eda1388f3178" containerID="e8ce4cccfe26d642777b11e968215b78f68040dbe9e56584f30858862a849a72" exitCode=0 Oct 30 16:47:45.731495 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.731388 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67785c878-8c9pr" event={"ID":"3794a50e-6c2f-40bc-9be7-eda1388f3178","Type":"ContainerDied","Data":"e8ce4cccfe26d642777b11e968215b78f68040dbe9e56584f30858862a849a72"} Oct 30 16:47:45.804606 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.804581 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:45.886710 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.886632 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-certificates\") pod \"3794a50e-6c2f-40bc-9be7-eda1388f3178\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " Oct 30 16:47:45.886710 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.886689 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3794a50e-6c2f-40bc-9be7-eda1388f3178-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3794a50e-6c2f-40bc-9be7-eda1388f3178-ca-trust-extracted\") pod \"3794a50e-6c2f-40bc-9be7-eda1388f3178\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " Oct 30 16:47:45.886941 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.886726 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-trusted-ca\") pod \"3794a50e-6c2f-40bc-9be7-eda1388f3178\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " Oct 30 16:47:45.886941 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.886779 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\") pod \"3794a50e-6c2f-40bc-9be7-eda1388f3178\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " Oct 30 16:47:45.886941 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.886824 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-installation-pull-secrets\") pod \"3794a50e-6c2f-40bc-9be7-eda1388f3178\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " Oct 30 16:47:45.886941 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.886878 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-kube-api-access-7xjxv\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-kube-api-access-7xjxv\") pod \"3794a50e-6c2f-40bc-9be7-eda1388f3178\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " Oct 30 16:47:45.887158 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.886950 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-bound-sa-token\") pod \"3794a50e-6c2f-40bc-9be7-eda1388f3178\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " Oct 30 16:47:45.887158 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.887029 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-image-registry-private-configuration\") pod \"3794a50e-6c2f-40bc-9be7-eda1388f3178\" (UID: \"3794a50e-6c2f-40bc-9be7-eda1388f3178\") " Oct 30 16:47:45.887158 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.887107 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-certificates" pod "3794a50e-6c2f-40bc-9be7-eda1388f3178" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:45.887279 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.887196 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-trusted-ca" pod "3794a50e-6c2f-40bc-9be7-eda1388f3178" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:45.887325 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.887307 4828 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-certificates\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:47:45.887385 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.887331 4828 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3794a50e-6c2f-40bc-9be7-eda1388f3178-trusted-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:47:45.889460 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.889408 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls" pod "3794a50e-6c2f-40bc-9be7-eda1388f3178" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:45.889774 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.889754 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-image-registry-private-configuration" pod "3794a50e-6c2f-40bc-9be7-eda1388f3178" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:45.889874 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.889825 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-kube-api-access-7xjxv" pod "3794a50e-6c2f-40bc-9be7-eda1388f3178" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178"). InnerVolumeSpecName "kube-api-access-7xjxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:45.889945 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.889885 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-bound-sa-token" pod "3794a50e-6c2f-40bc-9be7-eda1388f3178" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:45.889945 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.889924 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-installation-pull-secrets" pod "3794a50e-6c2f-40bc-9be7-eda1388f3178" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:45.893776 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.893753 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3794a50e-6c2f-40bc-9be7-eda1388f3178-ca-trust-extracted" pod "3794a50e-6c2f-40bc-9be7-eda1388f3178" (UID: "3794a50e-6c2f-40bc-9be7-eda1388f3178"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:45.988642 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.988603 4828 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-bound-sa-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:47:45.988642 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.988637 4828 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-image-registry-private-configuration\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:47:45.988642 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.988647 4828 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3794a50e-6c2f-40bc-9be7-eda1388f3178-ca-trust-extracted\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:47:45.988867 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.988657 4828 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-registry-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:47:45.988867 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.988665 4828 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3794a50e-6c2f-40bc-9be7-eda1388f3178-installation-pull-secrets\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:47:45.988867 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:45.988675 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xjxv\" (UniqueName: \"kubernetes.io/projected/3794a50e-6c2f-40bc-9be7-eda1388f3178-kube-api-access-7xjxv\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:47:46.500025 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:46.499991 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:46.650343 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:46.650304 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:46.737277 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:46.737243 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67785c878-8c9pr" Oct 30 16:47:46.737277 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:46.737258 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67785c878-8c9pr" event={"ID":"3794a50e-6c2f-40bc-9be7-eda1388f3178","Type":"ContainerDied","Data":"881ae6a0e628e80ae92e2e943cf669a586842320ace0ac3ba5d83f7f90d307b5"} Oct 30 16:47:46.737517 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:46.737324 4828 scope.go:122] "RemoveContainer" containerID="e8ce4cccfe26d642777b11e968215b78f68040dbe9e56584f30858862a849a72" Oct 30 16:47:46.809669 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:46.809629 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67785c878-8c9pr"] Oct 30 16:47:46.822115 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:46.822081 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67785c878-8c9pr"] Oct 30 16:47:48.487323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:48.487285 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3794a50e-6c2f-40bc-9be7-eda1388f3178" path="/var/lib/kubelet/pods/3794a50e-6c2f-40bc-9be7-eda1388f3178/volumes" Oct 30 16:47:50.695954 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:50.695917 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:50.842576 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:50.842537 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:52.386802 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:52.386766 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:52.575254 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:52.575214 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:57.795571 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:57.795533 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:57.950789 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:47:57.950756 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:00.979202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:00.979159 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:01.129883 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:01.129842 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:07.908209 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:48:07.908170 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:48:08.356637 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.356598 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" podUID="d38312b4-bd6e-4a5f-8a33-d93fb827debc" containerName="registry" containerID="cri-o://7a382cc70e2ec4154df1cbb4f510025c8c98b806b6b3e0b963d3136b40329edf" gracePeriod=30 Oct 30 16:48:08.599116 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.599093 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:48:08.773577 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.773545 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-trusted-ca\") pod \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " Oct 30 16:48:08.773731 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.773587 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d38312b4-bd6e-4a5f-8a33-d93fb827debc-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d38312b4-bd6e-4a5f-8a33-d93fb827debc-ca-trust-extracted\") pod \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " Oct 30 16:48:08.773731 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.773643 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-kube-api-access-q52w4\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-kube-api-access-q52w4\") pod \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " Oct 30 16:48:08.773731 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.773676 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\") pod \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " Oct 30 16:48:08.773731 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.773713 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-bound-sa-token\") pod \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " Oct 30 16:48:08.773980 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.773799 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-image-registry-private-configuration\") pod \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " Oct 30 16:48:08.774082 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.773974 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-certificates\") pod \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " Oct 30 16:48:08.774082 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.774014 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-installation-pull-secrets\") pod \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\" (UID: \"d38312b4-bd6e-4a5f-8a33-d93fb827debc\") " Oct 30 16:48:08.774082 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.774041 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-trusted-ca" pod "d38312b4-bd6e-4a5f-8a33-d93fb827debc" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:08.774235 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.774217 4828 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-trusted-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:48:08.774748 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.774702 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-certificates" pod "d38312b4-bd6e-4a5f-8a33-d93fb827debc" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:08.776644 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.776595 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-image-registry-private-configuration" pod "d38312b4-bd6e-4a5f-8a33-d93fb827debc" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:48:08.776644 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.776590 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-bound-sa-token" pod "d38312b4-bd6e-4a5f-8a33-d93fb827debc" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:08.776800 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.776665 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls" pod "d38312b4-bd6e-4a5f-8a33-d93fb827debc" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:08.776800 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.776688 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-kube-api-access-q52w4" pod "d38312b4-bd6e-4a5f-8a33-d93fb827debc" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc"). InnerVolumeSpecName "kube-api-access-q52w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:08.776883 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.776864 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-installation-pull-secrets" pod "d38312b4-bd6e-4a5f-8a33-d93fb827debc" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:48:08.780942 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.780862 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d38312b4-bd6e-4a5f-8a33-d93fb827debc-ca-trust-extracted" pod "d38312b4-bd6e-4a5f-8a33-d93fb827debc" (UID: "d38312b4-bd6e-4a5f-8a33-d93fb827debc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:08.821854 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.821818 4828 generic.go:356] "Generic (PLEG): container finished" podID="d38312b4-bd6e-4a5f-8a33-d93fb827debc" containerID="7a382cc70e2ec4154df1cbb4f510025c8c98b806b6b3e0b963d3136b40329edf" exitCode=0 Oct 30 16:48:08.822021 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.821869 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" Oct 30 16:48:08.822021 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.821888 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" event={"ID":"d38312b4-bd6e-4a5f-8a33-d93fb827debc","Type":"ContainerDied","Data":"7a382cc70e2ec4154df1cbb4f510025c8c98b806b6b3e0b963d3136b40329edf"} Oct 30 16:48:08.822021 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.821953 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797d7cb6cf-rzmrh" event={"ID":"d38312b4-bd6e-4a5f-8a33-d93fb827debc","Type":"ContainerDied","Data":"ae151e4f05975668309df9c14f8bcfca724c4682c2bb60814b7a8fde4905800a"} Oct 30 16:48:08.822021 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.821983 4828 scope.go:122] "RemoveContainer" containerID="7a382cc70e2ec4154df1cbb4f510025c8c98b806b6b3e0b963d3136b40329edf" Oct 30 16:48:08.823592 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.823565 4828 generic.go:356] "Generic (PLEG): container finished" podID="b516f174-836e-4711-8999-80ce5a592e28" containerID="4c74f557998eabc62244dafb8eefeb0215c38c04a62d9f7eadcd7d8adcc70519" exitCode=0 Oct 30 16:48:08.823721 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.823625 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jmx89" event={"ID":"b516f174-836e-4711-8999-80ce5a592e28","Type":"ContainerDied","Data":"4c74f557998eabc62244dafb8eefeb0215c38c04a62d9f7eadcd7d8adcc70519"} Oct 30 16:48:08.823930 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.823895 4828 scope.go:122] "RemoveContainer" containerID="4c74f557998eabc62244dafb8eefeb0215c38c04a62d9f7eadcd7d8adcc70519" Oct 30 16:48:08.830993 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.830971 4828 scope.go:122] "RemoveContainer" containerID="7a382cc70e2ec4154df1cbb4f510025c8c98b806b6b3e0b963d3136b40329edf" Oct 30 16:48:08.831423 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:48:08.831284 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a382cc70e2ec4154df1cbb4f510025c8c98b806b6b3e0b963d3136b40329edf\": container with ID starting with 7a382cc70e2ec4154df1cbb4f510025c8c98b806b6b3e0b963d3136b40329edf not found: ID does not exist" containerID="7a382cc70e2ec4154df1cbb4f510025c8c98b806b6b3e0b963d3136b40329edf" Oct 30 16:48:08.831423 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.831318 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a382cc70e2ec4154df1cbb4f510025c8c98b806b6b3e0b963d3136b40329edf"} err="failed to get container status \"7a382cc70e2ec4154df1cbb4f510025c8c98b806b6b3e0b963d3136b40329edf\": rpc error: code = NotFound desc = could not find container \"7a382cc70e2ec4154df1cbb4f510025c8c98b806b6b3e0b963d3136b40329edf\": container with ID starting with 7a382cc70e2ec4154df1cbb4f510025c8c98b806b6b3e0b963d3136b40329edf not found: ID does not exist" Oct 30 16:48:08.874684 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.874657 4828 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d38312b4-bd6e-4a5f-8a33-d93fb827debc-ca-trust-extracted\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:48:08.874684 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.874682 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q52w4\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-kube-api-access-q52w4\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:48:08.874860 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.874693 4828 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:48:08.874860 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.874702 4828 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d38312b4-bd6e-4a5f-8a33-d93fb827debc-bound-sa-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:48:08.874860 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.874711 4828 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-image-registry-private-configuration\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:48:08.874860 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.874721 4828 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d38312b4-bd6e-4a5f-8a33-d93fb827debc-registry-certificates\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:48:08.874860 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.874729 4828 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d38312b4-bd6e-4a5f-8a33-d93fb827debc-installation-pull-secrets\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:48:08.925140 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.925105 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-797d7cb6cf-rzmrh"] Oct 30 16:48:08.929145 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:08.929123 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-797d7cb6cf-rzmrh"] Oct 30 16:48:09.672074 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.672040 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rdx52-2z9bt"] Oct 30 16:48:09.672334 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.672322 4828 cpu_manager.go:414] "containerMap: removing container" podUID="3794a50e-6c2f-40bc-9be7-eda1388f3178" containerName="registry" Oct 30 16:48:09.672379 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.672335 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3794a50e-6c2f-40bc-9be7-eda1388f3178" containerName="registry" Oct 30 16:48:09.672379 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.672353 4828 cpu_manager.go:414] "containerMap: removing container" podUID="d38312b4-bd6e-4a5f-8a33-d93fb827debc" containerName="registry" Oct 30 16:48:09.672379 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.672358 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38312b4-bd6e-4a5f-8a33-d93fb827debc" containerName="registry" Oct 30 16:48:09.672461 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.672404 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="3794a50e-6c2f-40bc-9be7-eda1388f3178" containerName="registry" Oct 30 16:48:09.672461 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.672416 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38312b4-bd6e-4a5f-8a33-d93fb827debc" containerName="registry" Oct 30 16:48:09.675573 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.675552 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:09.696154 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.696121 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rdx52-2z9bt"] Oct 30 16:48:09.782232 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.782193 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/699017b4-6198-43f6-94aa-6cb059303c10-archives-path\") pod \"periodic-gathering-rdx52-2z9bt\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:09.782232 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.782235 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-service-ca-bundle\") pod \"periodic-gathering-rdx52-2z9bt\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:09.782476 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.782334 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-trusted-ca-bundle\") pod \"periodic-gathering-rdx52-2z9bt\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:09.782476 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.782380 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjvz9\" (UniqueName: \"kubernetes.io/projected/699017b4-6198-43f6-94aa-6cb059303c10-kube-api-access-wjvz9\") pod \"periodic-gathering-rdx52-2z9bt\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:09.830347 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.830310 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jmx89" event={"ID":"b516f174-836e-4711-8999-80ce5a592e28","Type":"ContainerStarted","Data":"e216b857fa200686e5f25edab1b3948b8b66a79deee7358f5c3382c2f6660791"} Oct 30 16:48:09.883059 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.883023 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/699017b4-6198-43f6-94aa-6cb059303c10-archives-path\") pod \"periodic-gathering-rdx52-2z9bt\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:09.883059 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.883062 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-service-ca-bundle\") pod \"periodic-gathering-rdx52-2z9bt\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:09.883313 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.883101 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-trusted-ca-bundle\") pod \"periodic-gathering-rdx52-2z9bt\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:09.883313 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.883136 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjvz9\" (UniqueName: \"kubernetes.io/projected/699017b4-6198-43f6-94aa-6cb059303c10-kube-api-access-wjvz9\") pod \"periodic-gathering-rdx52-2z9bt\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:09.883514 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.883488 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/699017b4-6198-43f6-94aa-6cb059303c10-archives-path\") pod \"periodic-gathering-rdx52-2z9bt\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:09.883774 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.883756 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-service-ca-bundle\") pod \"periodic-gathering-rdx52-2z9bt\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:09.884129 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.884111 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-trusted-ca-bundle\") pod \"periodic-gathering-rdx52-2z9bt\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:09.910788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.910754 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjvz9\" (UniqueName: \"kubernetes.io/projected/699017b4-6198-43f6-94aa-6cb059303c10-kube-api-access-wjvz9\") pod \"periodic-gathering-rdx52-2z9bt\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:09.986642 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:09.986563 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:48:10.280701 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:10.280603 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rdx52-2z9bt"] Oct 30 16:48:10.282343 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:48:10.282317 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699017b4_6198_43f6_94aa_6cb059303c10.slice/crio-4096ba525bced7d94b158189d93eb7663ac11e50989690fe001ce53b6bcb63b8 WatchSource:0}: Error finding container 4096ba525bced7d94b158189d93eb7663ac11e50989690fe001ce53b6bcb63b8: Status 404 returned error can't find the container with id 4096ba525bced7d94b158189d93eb7663ac11e50989690fe001ce53b6bcb63b8 Oct 30 16:48:10.490374 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:10.490332 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d38312b4-bd6e-4a5f-8a33-d93fb827debc" path="/var/lib/kubelet/pods/d38312b4-bd6e-4a5f-8a33-d93fb827debc/volumes" Oct 30 16:48:10.835995 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:10.835962 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rdx52-2z9bt" event={"ID":"699017b4-6198-43f6-94aa-6cb059303c10","Type":"ContainerStarted","Data":"6734033d0c9db2da4623f26f1937c00cc357cae163d78ec2d86193422ee98594"} Oct 30 16:48:10.836167 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:10.836011 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rdx52-2z9bt" event={"ID":"699017b4-6198-43f6-94aa-6cb059303c10","Type":"ContainerStarted","Data":"4096ba525bced7d94b158189d93eb7663ac11e50989690fe001ce53b6bcb63b8"} Oct 30 16:48:10.878707 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:10.878656 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rdx52-2z9bt" podStartSLOduration=1.8786422379999999 podStartE2EDuration="1.878642238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:10.878129967 +0000 UTC m=+68.523640147" watchObservedRunningTime="2026-10-30 16:48:10.878642238 +0000 UTC m=+68.524152418" Oct 30 16:48:12.802255 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:12.802220 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:12.843572 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:12.843542 4828 generic.go:356] "Generic (PLEG): container finished" podID="11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e" containerID="0ea948261e59c5dc96e8264981118d5b3947060bc0a50af3b0f5c640645c1df0" exitCode=0 Oct 30 16:48:12.843700 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:12.843611 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" event={"ID":"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e","Type":"ContainerDied","Data":"0ea948261e59c5dc96e8264981118d5b3947060bc0a50af3b0f5c640645c1df0"} Oct 30 16:48:12.843935 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:12.843916 4828 scope.go:122] "RemoveContainer" containerID="0ea948261e59c5dc96e8264981118d5b3947060bc0a50af3b0f5c640645c1df0" Oct 30 16:48:12.956594 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:12.956559 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:13.282790 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:13.282750 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:13.428501 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:13.428460 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:13.849708 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:13.849671 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kzh45" event={"ID":"11fe6a2d-ed34-40cb-9a73-a637bd3a2d6e","Type":"ContainerStarted","Data":"05bdcb86b311dcf03a33f24e500bb315ca035b618c0db14dcd7962ef2e1d422b"} Oct 30 16:48:13.997793 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:13.997755 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:14.261813 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:14.261787 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:14.784126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:14.784096 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:14.932467 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:14.932433 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:15.083542 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:15.083463 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:17.573762 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:17.573730 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:17.731332 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:17.731301 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:27.813780 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:27.813736 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:27.968362 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:27.968327 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:32.869237 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:48:32.869197 4828 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a14012_73eb_4f3b_b762_f56c6a963e44.slice/crio-071604208a7d920227d29eaa1f76504d152027748a890939553f726b9fae546a.scope\": RecentStats: unable to find data in memory cache]" Oct 30 16:48:32.932029 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:32.931998 4828 generic.go:356] "Generic (PLEG): container finished" podID="35a14012-73eb-4f3b-b762-f56c6a963e44" containerID="071604208a7d920227d29eaa1f76504d152027748a890939553f726b9fae546a" exitCode=0 Oct 30 16:48:32.932194 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:32.932064 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" event={"ID":"35a14012-73eb-4f3b-b762-f56c6a963e44","Type":"ContainerDied","Data":"071604208a7d920227d29eaa1f76504d152027748a890939553f726b9fae546a"} Oct 30 16:48:32.932420 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:32.932403 4828 scope.go:122] "RemoveContainer" containerID="071604208a7d920227d29eaa1f76504d152027748a890939553f726b9fae546a" Oct 30 16:48:33.937388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:33.937350 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2k2pq" event={"ID":"35a14012-73eb-4f3b-b762-f56c6a963e44","Type":"ContainerStarted","Data":"a95549ca4f81cda7fa2275f919bdc135810edbaf59530f3b2d665dce0f7b42b8"} Oct 30 16:48:37.909466 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:48:37.909432 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:48:52.481150 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:48:52.481113 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:07.908484 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:49:07.908445 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:49:34.149971 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:34.149929 4828 generic.go:356] "Generic (PLEG): container finished" podID="699017b4-6198-43f6-94aa-6cb059303c10" containerID="6734033d0c9db2da4623f26f1937c00cc357cae163d78ec2d86193422ee98594" exitCode=0 Oct 30 16:49:34.150488 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:34.149988 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rdx52-2z9bt" event={"ID":"699017b4-6198-43f6-94aa-6cb059303c10","Type":"ContainerDied","Data":"6734033d0c9db2da4623f26f1937c00cc357cae163d78ec2d86193422ee98594"} Oct 30 16:49:35.272755 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:35.272729 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:49:35.338943 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:35.338886 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699017b4-6198-43f6-94aa-6cb059303c10-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/699017b4-6198-43f6-94aa-6cb059303c10-archives-path\") pod \"699017b4-6198-43f6-94aa-6cb059303c10\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " Oct 30 16:49:35.338943 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:35.338946 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699017b4-6198-43f6-94aa-6cb059303c10-kube-api-access-wjvz9\" (UniqueName: \"kubernetes.io/projected/699017b4-6198-43f6-94aa-6cb059303c10-kube-api-access-wjvz9\") pod \"699017b4-6198-43f6-94aa-6cb059303c10\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " Oct 30 16:49:35.339192 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:35.338968 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-service-ca-bundle\") pod \"699017b4-6198-43f6-94aa-6cb059303c10\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " Oct 30 16:49:35.339192 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:35.339023 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-trusted-ca-bundle\") pod \"699017b4-6198-43f6-94aa-6cb059303c10\" (UID: \"699017b4-6198-43f6-94aa-6cb059303c10\") " Oct 30 16:49:35.339364 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:35.339334 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699017b4-6198-43f6-94aa-6cb059303c10-archives-path" pod "699017b4-6198-43f6-94aa-6cb059303c10" (UID: "699017b4-6198-43f6-94aa-6cb059303c10"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.339427 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:35.339361 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-service-ca-bundle" pod "699017b4-6198-43f6-94aa-6cb059303c10" (UID: "699017b4-6198-43f6-94aa-6cb059303c10"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:35.339427 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:35.339392 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-trusted-ca-bundle" pod "699017b4-6198-43f6-94aa-6cb059303c10" (UID: "699017b4-6198-43f6-94aa-6cb059303c10"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:35.341264 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:35.341244 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699017b4-6198-43f6-94aa-6cb059303c10-kube-api-access-wjvz9" pod "699017b4-6198-43f6-94aa-6cb059303c10" (UID: "699017b4-6198-43f6-94aa-6cb059303c10"). InnerVolumeSpecName "kube-api-access-wjvz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:35.440138 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:35.440056 4828 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-trusted-ca-bundle\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.440138 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:35.440086 4828 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/699017b4-6198-43f6-94aa-6cb059303c10-archives-path\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.440138 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:35.440097 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjvz9\" (UniqueName: \"kubernetes.io/projected/699017b4-6198-43f6-94aa-6cb059303c10-kube-api-access-wjvz9\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.440138 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:35.440106 4828 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/699017b4-6198-43f6-94aa-6cb059303c10-service-ca-bundle\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:49:36.159047 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:36.159012 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rdx52-2z9bt" event={"ID":"699017b4-6198-43f6-94aa-6cb059303c10","Type":"ContainerDied","Data":"4096ba525bced7d94b158189d93eb7663ac11e50989690fe001ce53b6bcb63b8"} Oct 30 16:49:36.159221 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:36.159050 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rdx52-2z9bt" Oct 30 16:49:36.159221 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:36.159062 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4096ba525bced7d94b158189d93eb7663ac11e50989690fe001ce53b6bcb63b8" Oct 30 16:49:36.662714 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:36.662671 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:36.831147 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:36.831113 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:37.908307 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:49:37.908272 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:49:52.654575 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:49:52.654539 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:05.692565 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:05.692521 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:07.909279 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:50:07.909234 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:50:09.743648 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:09.743601 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:09.893909 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:09.893875 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:16.959391 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:16.959354 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:17.118367 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.118330 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:17.283889 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.283851 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:17.842187 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.842147 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5"] Oct 30 16:50:17.842519 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.842500 4828 cpu_manager.go:414] "containerMap: removing container" podUID="699017b4-6198-43f6-94aa-6cb059303c10" containerName="insights-gathering" Oct 30 16:50:17.842606 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.842521 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699017b4-6198-43f6-94aa-6cb059303c10" containerName="insights-gathering" Oct 30 16:50:17.842667 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.842624 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="699017b4-6198-43f6-94aa-6cb059303c10" containerName="insights-gathering" Oct 30 16:50:17.845656 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.845635 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" Oct 30 16:50:17.864734 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.864710 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:50:17.878365 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.878337 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:50:17.878516 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.878454 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-6lvt5\"" Oct 30 16:50:17.879617 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.879593 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5"] Oct 30 16:50:17.923579 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.923540 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-262mm\" (UniqueName: \"kubernetes.io/projected/c4f485fd-edee-4c49-a315-fa1e1809d60c-kube-api-access-262mm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5\" (UID: \"c4f485fd-edee-4c49-a315-fa1e1809d60c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" Oct 30 16:50:17.923755 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.923609 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5\" (UID: \"c4f485fd-edee-4c49-a315-fa1e1809d60c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" Oct 30 16:50:17.923755 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:17.923665 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5\" (UID: \"c4f485fd-edee-4c49-a315-fa1e1809d60c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" Oct 30 16:50:18.024577 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:18.024544 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5\" (UID: \"c4f485fd-edee-4c49-a315-fa1e1809d60c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" Oct 30 16:50:18.024955 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:18.024586 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5\" (UID: \"c4f485fd-edee-4c49-a315-fa1e1809d60c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" Oct 30 16:50:18.024955 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:18.024639 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-262mm\" (UniqueName: \"kubernetes.io/projected/c4f485fd-edee-4c49-a315-fa1e1809d60c-kube-api-access-262mm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5\" (UID: \"c4f485fd-edee-4c49-a315-fa1e1809d60c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" Oct 30 16:50:18.025049 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:18.025029 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5\" (UID: \"c4f485fd-edee-4c49-a315-fa1e1809d60c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" Oct 30 16:50:18.025089 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:18.025037 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5\" (UID: \"c4f485fd-edee-4c49-a315-fa1e1809d60c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" Oct 30 16:50:18.049995 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:18.049966 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-262mm\" (UniqueName: \"kubernetes.io/projected/c4f485fd-edee-4c49-a315-fa1e1809d60c-kube-api-access-262mm\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5\" (UID: \"c4f485fd-edee-4c49-a315-fa1e1809d60c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" Oct 30 16:50:18.161525 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:18.161437 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" Oct 30 16:50:18.458688 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:18.458660 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5"] Oct 30 16:50:18.461246 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:50:18.461217 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f485fd_edee_4c49_a315_fa1e1809d60c.slice/crio-dcfdee87032bb55dfefc3672fdd2e5e6b749b66bed8fd71d8efd74c3bc90117e WatchSource:0}: Error finding container dcfdee87032bb55dfefc3672fdd2e5e6b749b66bed8fd71d8efd74c3bc90117e: Status 404 returned error can't find the container with id dcfdee87032bb55dfefc3672fdd2e5e6b749b66bed8fd71d8efd74c3bc90117e Oct 30 16:50:19.310684 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:19.310649 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" event={"ID":"c4f485fd-edee-4c49-a315-fa1e1809d60c","Type":"ContainerStarted","Data":"dcfdee87032bb55dfefc3672fdd2e5e6b749b66bed8fd71d8efd74c3bc90117e"} Oct 30 16:50:24.331539 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:24.331501 4828 generic.go:356] "Generic (PLEG): container finished" podID="c4f485fd-edee-4c49-a315-fa1e1809d60c" containerID="73a170a4b78a1af1d17334eb361ebc494d96661f65ffab26fdd3be168c286077" exitCode=0 Oct 30 16:50:24.331963 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:24.331547 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" event={"ID":"c4f485fd-edee-4c49-a315-fa1e1809d60c","Type":"ContainerDied","Data":"73a170a4b78a1af1d17334eb361ebc494d96661f65ffab26fdd3be168c286077"} Oct 30 16:50:26.341566 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:26.341533 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" event={"ID":"c4f485fd-edee-4c49-a315-fa1e1809d60c","Type":"ContainerStarted","Data":"94fe274d835ff9acbfcf36afedc41244da17d836c5b8f5b70c309f203598b130"} Oct 30 16:50:27.347469 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:27.347432 4828 generic.go:356] "Generic (PLEG): container finished" podID="c4f485fd-edee-4c49-a315-fa1e1809d60c" containerID="94fe274d835ff9acbfcf36afedc41244da17d836c5b8f5b70c309f203598b130" exitCode=0 Oct 30 16:50:27.347882 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:27.347511 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" event={"ID":"c4f485fd-edee-4c49-a315-fa1e1809d60c","Type":"ContainerDied","Data":"94fe274d835ff9acbfcf36afedc41244da17d836c5b8f5b70c309f203598b130"} Oct 30 16:50:33.373121 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:33.373079 4828 generic.go:356] "Generic (PLEG): container finished" podID="c4f485fd-edee-4c49-a315-fa1e1809d60c" containerID="6416f1090943038dee8d5a03a37e0caea5ee42743777592407b1ca380da072ea" exitCode=0 Oct 30 16:50:33.373559 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:33.373121 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" event={"ID":"c4f485fd-edee-4c49-a315-fa1e1809d60c","Type":"ContainerDied","Data":"6416f1090943038dee8d5a03a37e0caea5ee42743777592407b1ca380da072ea"} Oct 30 16:50:34.498816 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:34.498790 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" Oct 30 16:50:34.563443 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:34.563411 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f485fd-edee-4c49-a315-fa1e1809d60c-kube-api-access-262mm\" (UniqueName: \"kubernetes.io/projected/c4f485fd-edee-4c49-a315-fa1e1809d60c-kube-api-access-262mm\") pod \"c4f485fd-edee-4c49-a315-fa1e1809d60c\" (UID: \"c4f485fd-edee-4c49-a315-fa1e1809d60c\") " Oct 30 16:50:34.563630 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:34.563524 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-bundle\") pod \"c4f485fd-edee-4c49-a315-fa1e1809d60c\" (UID: \"c4f485fd-edee-4c49-a315-fa1e1809d60c\") " Oct 30 16:50:34.563630 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:34.563564 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-util\" (UniqueName: \"kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-util\") pod \"c4f485fd-edee-4c49-a315-fa1e1809d60c\" (UID: \"c4f485fd-edee-4c49-a315-fa1e1809d60c\") " Oct 30 16:50:34.564139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:34.564110 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-bundle" pod "c4f485fd-edee-4c49-a315-fa1e1809d60c" (UID: "c4f485fd-edee-4c49-a315-fa1e1809d60c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:34.565777 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:34.565742 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f485fd-edee-4c49-a315-fa1e1809d60c-kube-api-access-262mm" pod "c4f485fd-edee-4c49-a315-fa1e1809d60c" (UID: "c4f485fd-edee-4c49-a315-fa1e1809d60c"). InnerVolumeSpecName "kube-api-access-262mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:34.568299 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:34.568269 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-util" pod "c4f485fd-edee-4c49-a315-fa1e1809d60c" (UID: "c4f485fd-edee-4c49-a315-fa1e1809d60c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:34.665030 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:34.664949 4828 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-bundle\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:50:34.665030 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:34.664976 4828 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4f485fd-edee-4c49-a315-fa1e1809d60c-util\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:50:34.665030 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:34.664985 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-262mm\" (UniqueName: \"kubernetes.io/projected/c4f485fd-edee-4c49-a315-fa1e1809d60c-kube-api-access-262mm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:50:35.382281 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:35.382244 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" event={"ID":"c4f485fd-edee-4c49-a315-fa1e1809d60c","Type":"ContainerDied","Data":"dcfdee87032bb55dfefc3672fdd2e5e6b749b66bed8fd71d8efd74c3bc90117e"} Oct 30 16:50:35.382281 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:35.382281 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cl9rp5" Oct 30 16:50:35.382478 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:35.382293 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcfdee87032bb55dfefc3672fdd2e5e6b749b66bed8fd71d8efd74c3bc90117e" Oct 30 16:50:37.908799 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:50:37.908756 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:50:39.159874 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:39.159815 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:40.496658 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:40.496625 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:41.096082 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.096048 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2"] Oct 30 16:50:41.096356 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.096343 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c4f485fd-edee-4c49-a315-fa1e1809d60c" containerName="extract" Oct 30 16:50:41.096434 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.096358 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f485fd-edee-4c49-a315-fa1e1809d60c" containerName="extract" Oct 30 16:50:41.096434 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.096374 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c4f485fd-edee-4c49-a315-fa1e1809d60c" containerName="util" Oct 30 16:50:41.096434 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.096379 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f485fd-edee-4c49-a315-fa1e1809d60c" containerName="util" Oct 30 16:50:41.096434 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.096391 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c4f485fd-edee-4c49-a315-fa1e1809d60c" containerName="pull" Oct 30 16:50:41.096434 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.096397 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f485fd-edee-4c49-a315-fa1e1809d60c" containerName="pull" Oct 30 16:50:41.096677 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.096452 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f485fd-edee-4c49-a315-fa1e1809d60c" containerName="extract" Oct 30 16:50:41.138360 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.138322 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2"] Oct 30 16:50:41.138519 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.138427 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2" Oct 30 16:50:41.155804 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.155774 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zchwf\"" Oct 30 16:50:41.155804 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.155802 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:50:41.155804 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.155818 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 16:50:41.213105 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.213069 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2gn\" (UniqueName: \"kubernetes.io/projected/faddc8c4-bf88-4457-b069-2d0bc985ee41-kube-api-access-xd2gn\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-kf7f2\" (UID: \"faddc8c4-bf88-4457-b069-2d0bc985ee41\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2" Oct 30 16:50:41.213105 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.213103 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faddc8c4-bf88-4457-b069-2d0bc985ee41-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-kf7f2\" (UID: \"faddc8c4-bf88-4457-b069-2d0bc985ee41\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2" Oct 30 16:50:41.314348 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.314312 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2gn\" (UniqueName: \"kubernetes.io/projected/faddc8c4-bf88-4457-b069-2d0bc985ee41-kube-api-access-xd2gn\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-kf7f2\" (UID: \"faddc8c4-bf88-4457-b069-2d0bc985ee41\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2" Oct 30 16:50:41.314348 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.314350 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faddc8c4-bf88-4457-b069-2d0bc985ee41-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-kf7f2\" (UID: \"faddc8c4-bf88-4457-b069-2d0bc985ee41\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2" Oct 30 16:50:41.314710 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.314693 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faddc8c4-bf88-4457-b069-2d0bc985ee41-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-kf7f2\" (UID: \"faddc8c4-bf88-4457-b069-2d0bc985ee41\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2" Oct 30 16:50:41.339713 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.339677 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2gn\" (UniqueName: \"kubernetes.io/projected/faddc8c4-bf88-4457-b069-2d0bc985ee41-kube-api-access-xd2gn\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-kf7f2\" (UID: \"faddc8c4-bf88-4457-b069-2d0bc985ee41\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2" Oct 30 16:50:41.461036 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.460957 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2" Oct 30 16:50:41.811198 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:41.811165 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2"] Oct 30 16:50:41.814055 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:50:41.814027 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaddc8c4_bf88_4457_b069_2d0bc985ee41.slice/crio-d8e805d7a98d167155993d74cfcc5ef6a9f707f79f55f896ca373d6aafe973f2 WatchSource:0}: Error finding container d8e805d7a98d167155993d74cfcc5ef6a9f707f79f55f896ca373d6aafe973f2: Status 404 returned error can't find the container with id d8e805d7a98d167155993d74cfcc5ef6a9f707f79f55f896ca373d6aafe973f2 Oct 30 16:50:42.348642 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:42.348606 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:42.409667 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:42.409620 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2" event={"ID":"faddc8c4-bf88-4457-b069-2d0bc985ee41","Type":"ContainerStarted","Data":"d8e805d7a98d167155993d74cfcc5ef6a9f707f79f55f896ca373d6aafe973f2"} Oct 30 16:50:42.502805 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:42.502764 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:44.419798 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:44.419760 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2" event={"ID":"faddc8c4-bf88-4457-b069-2d0bc985ee41","Type":"ContainerStarted","Data":"5b160ff4394e592638c19ba7c384e66f4c81b2d17875585e5c5feff387b85edb"} Oct 30 16:50:44.501427 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:44.501373 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-kf7f2" podStartSLOduration=1.5141723649999999 podStartE2EDuration="3.501354575s" totalImagesPullingTime="1.98718221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:44.486917797 +0000 UTC m=+222.132427968" watchObservedRunningTime="2026-10-30 16:50:44.501354575 +0000 UTC m=+222.146864757" Oct 30 16:50:44.822655 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:44.822622 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:44.997272 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:44.997239 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:45.172113 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.172014 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:45.738540 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.738499 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh"] Oct 30 16:50:45.756984 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.756958 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" Oct 30 16:50:45.764151 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.764125 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh"] Oct 30 16:50:45.776480 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.776453 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-mv62f\"" Oct 30 16:50:45.776480 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.776476 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:50:45.776657 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.776488 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:50:45.852753 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.852713 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgthf\" (UniqueName: \"kubernetes.io/projected/a4096055-d374-4a42-8719-6c5037934faf-kube-api-access-cgthf\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh\" (UID: \"a4096055-d374-4a42-8719-6c5037934faf\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" Oct 30 16:50:45.852970 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.852830 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh\" (UID: \"a4096055-d374-4a42-8719-6c5037934faf\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" Oct 30 16:50:45.852970 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.852929 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh\" (UID: \"a4096055-d374-4a42-8719-6c5037934faf\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" Oct 30 16:50:45.954065 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.954025 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh\" (UID: \"a4096055-d374-4a42-8719-6c5037934faf\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" Oct 30 16:50:45.954065 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.954068 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh\" (UID: \"a4096055-d374-4a42-8719-6c5037934faf\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" Oct 30 16:50:45.954275 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.954099 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgthf\" (UniqueName: \"kubernetes.io/projected/a4096055-d374-4a42-8719-6c5037934faf-kube-api-access-cgthf\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh\" (UID: \"a4096055-d374-4a42-8719-6c5037934faf\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" Oct 30 16:50:45.954473 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.954449 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh\" (UID: \"a4096055-d374-4a42-8719-6c5037934faf\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" Oct 30 16:50:45.954518 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.954457 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh\" (UID: \"a4096055-d374-4a42-8719-6c5037934faf\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" Oct 30 16:50:45.978855 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:45.978817 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgthf\" (UniqueName: \"kubernetes.io/projected/a4096055-d374-4a42-8719-6c5037934faf-kube-api-access-cgthf\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh\" (UID: \"a4096055-d374-4a42-8719-6c5037934faf\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" Oct 30 16:50:46.108014 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:46.107973 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" Oct 30 16:50:46.424217 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:46.424191 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh"] Oct 30 16:50:46.426149 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:50:46.426114 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4096055_d374_4a42_8719_6c5037934faf.slice/crio-ebd31e50d493d487a08dd1bbb5208e95d2150777f8075adac0ab4f26b904864a WatchSource:0}: Error finding container ebd31e50d493d487a08dd1bbb5208e95d2150777f8075adac0ab4f26b904864a: Status 404 returned error can't find the container with id ebd31e50d493d487a08dd1bbb5208e95d2150777f8075adac0ab4f26b904864a Oct 30 16:50:47.092646 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.092600 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:47.252802 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.252708 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89"] Oct 30 16:50:47.255186 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.255163 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89" Oct 30 16:50:47.275529 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.275507 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 16:50:47.276483 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.276466 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 16:50:47.287490 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.287454 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7hj7q\"" Oct 30 16:50:47.288723 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.288700 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89"] Oct 30 16:50:47.365309 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.365275 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c28fed5-a471-463c-bf9d-7c961968500c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-trl89\" (UID: \"5c28fed5-a471-463c-bf9d-7c961968500c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89" Oct 30 16:50:47.365476 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.365317 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkz69\" (UniqueName: \"kubernetes.io/projected/5c28fed5-a471-463c-bf9d-7c961968500c-kube-api-access-mkz69\") pod \"cert-manager-cainjector-5c57bbd4fc-trl89\" (UID: \"5c28fed5-a471-463c-bf9d-7c961968500c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89" Oct 30 16:50:47.433468 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.433439 4828 generic.go:356] "Generic (PLEG): container finished" podID="a4096055-d374-4a42-8719-6c5037934faf" containerID="7c312ee51fe65b8c2dde5f9dd053e77ae66d6600c3d5e9b958ed46a5e5c7899c" exitCode=0 Oct 30 16:50:47.433658 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.433535 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" event={"ID":"a4096055-d374-4a42-8719-6c5037934faf","Type":"ContainerDied","Data":"7c312ee51fe65b8c2dde5f9dd053e77ae66d6600c3d5e9b958ed46a5e5c7899c"} Oct 30 16:50:47.433658 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.433584 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" event={"ID":"a4096055-d374-4a42-8719-6c5037934faf","Type":"ContainerStarted","Data":"ebd31e50d493d487a08dd1bbb5208e95d2150777f8075adac0ab4f26b904864a"} Oct 30 16:50:47.466207 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.466172 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c28fed5-a471-463c-bf9d-7c961968500c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-trl89\" (UID: \"5c28fed5-a471-463c-bf9d-7c961968500c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89" Oct 30 16:50:47.466207 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.466207 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkz69\" (UniqueName: \"kubernetes.io/projected/5c28fed5-a471-463c-bf9d-7c961968500c-kube-api-access-mkz69\") pod \"cert-manager-cainjector-5c57bbd4fc-trl89\" (UID: \"5c28fed5-a471-463c-bf9d-7c961968500c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89" Oct 30 16:50:47.490872 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.490834 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkz69\" (UniqueName: \"kubernetes.io/projected/5c28fed5-a471-463c-bf9d-7c961968500c-kube-api-access-mkz69\") pod \"cert-manager-cainjector-5c57bbd4fc-trl89\" (UID: \"5c28fed5-a471-463c-bf9d-7c961968500c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89" Oct 30 16:50:47.491178 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.491159 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c28fed5-a471-463c-bf9d-7c961968500c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-trl89\" (UID: \"5c28fed5-a471-463c-bf9d-7c961968500c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89" Oct 30 16:50:47.494590 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.494560 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:47.579554 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.579513 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89" Oct 30 16:50:47.873279 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:47.873253 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89"] Oct 30 16:50:47.875281 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:50:47.875251 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c28fed5_a471_463c_bf9d_7c961968500c.slice/crio-5e029d0d763f1a96e991a2392035ec0ebaa54f07063d43aaf95c82628b85304e WatchSource:0}: Error finding container 5e029d0d763f1a96e991a2392035ec0ebaa54f07063d43aaf95c82628b85304e: Status 404 returned error can't find the container with id 5e029d0d763f1a96e991a2392035ec0ebaa54f07063d43aaf95c82628b85304e Oct 30 16:50:48.439204 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:48.439169 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89" event={"ID":"5c28fed5-a471-463c-bf9d-7c961968500c","Type":"ContainerStarted","Data":"5e029d0d763f1a96e991a2392035ec0ebaa54f07063d43aaf95c82628b85304e"} Oct 30 16:50:49.558078 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:49.557976 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:49.712038 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:49.711992 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:50.448502 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:50.448466 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89" event={"ID":"5c28fed5-a471-463c-bf9d-7c961968500c","Type":"ContainerStarted","Data":"ef11dc1c994d229ddee813d3946f6971c0cd86db62e1a3c191b441fb91011685"} Oct 30 16:50:50.492564 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:50.492512 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-trl89" podStartSLOduration=1.085734752 podStartE2EDuration="3.492497946s" totalImagesPullingTime="2.406763194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:50.491711999 +0000 UTC m=+228.137222180" watchObservedRunningTime="2026-10-30 16:50:50.492497946 +0000 UTC m=+228.138008125" Oct 30 16:50:52.711675 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:52.711629 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:55.493016 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:55.492971 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:56.474531 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:56.474437 4828 generic.go:356] "Generic (PLEG): container finished" podID="a4096055-d374-4a42-8719-6c5037934faf" containerID="0003e96ad07c2089936553a43b0f1b738b1416eb871fdb5de61e70ba5f99f75b" exitCode=0 Oct 30 16:50:56.474531 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:56.474473 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" event={"ID":"a4096055-d374-4a42-8719-6c5037934faf","Type":"ContainerDied","Data":"0003e96ad07c2089936553a43b0f1b738b1416eb871fdb5de61e70ba5f99f75b"} Oct 30 16:50:56.815189 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:56.815158 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:56.965788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:56.965747 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:50:57.480537 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:57.480503 4828 generic.go:356] "Generic (PLEG): container finished" podID="a4096055-d374-4a42-8719-6c5037934faf" containerID="aebf0bfa4077ba62af28a1e2bf156d4e74fb76ecfb646ba75ce9fc6e4eb9a831" exitCode=0 Oct 30 16:50:57.480537 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:57.480539 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" event={"ID":"a4096055-d374-4a42-8719-6c5037934faf","Type":"ContainerDied","Data":"aebf0bfa4077ba62af28a1e2bf156d4e74fb76ecfb646ba75ce9fc6e4eb9a831"} Oct 30 16:50:58.600396 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:58.600373 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" Oct 30 16:50:58.766696 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:58.766668 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-util\" (UniqueName: \"kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-util\") pod \"a4096055-d374-4a42-8719-6c5037934faf\" (UID: \"a4096055-d374-4a42-8719-6c5037934faf\") " Oct 30 16:50:58.766829 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:58.766816 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-bundle\") pod \"a4096055-d374-4a42-8719-6c5037934faf\" (UID: \"a4096055-d374-4a42-8719-6c5037934faf\") " Oct 30 16:50:58.766887 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:58.766852 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4096055-d374-4a42-8719-6c5037934faf-kube-api-access-cgthf\" (UniqueName: \"kubernetes.io/projected/a4096055-d374-4a42-8719-6c5037934faf-kube-api-access-cgthf\") pod \"a4096055-d374-4a42-8719-6c5037934faf\" (UID: \"a4096055-d374-4a42-8719-6c5037934faf\") " Oct 30 16:50:58.767205 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:58.767179 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-bundle" pod "a4096055-d374-4a42-8719-6c5037934faf" (UID: "a4096055-d374-4a42-8719-6c5037934faf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:58.769026 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:58.768993 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4096055-d374-4a42-8719-6c5037934faf-kube-api-access-cgthf" pod "a4096055-d374-4a42-8719-6c5037934faf" (UID: "a4096055-d374-4a42-8719-6c5037934faf"). InnerVolumeSpecName "kube-api-access-cgthf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:58.771283 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:58.771260 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-util" pod "a4096055-d374-4a42-8719-6c5037934faf" (UID: "a4096055-d374-4a42-8719-6c5037934faf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:58.867442 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:58.867406 4828 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-bundle\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:50:58.867442 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:58.867437 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgthf\" (UniqueName: \"kubernetes.io/projected/a4096055-d374-4a42-8719-6c5037934faf-kube-api-access-cgthf\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:50:58.867442 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:58.867447 4828 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4096055-d374-4a42-8719-6c5037934faf-util\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:50:59.491453 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:59.491418 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" Oct 30 16:50:59.491616 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:59.491422 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85vrqqh" event={"ID":"a4096055-d374-4a42-8719-6c5037934faf","Type":"ContainerDied","Data":"ebd31e50d493d487a08dd1bbb5208e95d2150777f8075adac0ab4f26b904864a"} Oct 30 16:50:59.491616 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:50:59.491557 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd31e50d493d487a08dd1bbb5208e95d2150777f8075adac0ab4f26b904864a" Oct 30 16:51:05.527409 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.527376 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-g226q"] Oct 30 16:51:05.527796 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.527682 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a4096055-d374-4a42-8719-6c5037934faf" containerName="extract" Oct 30 16:51:05.527796 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.527694 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4096055-d374-4a42-8719-6c5037934faf" containerName="extract" Oct 30 16:51:05.527796 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.527707 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a4096055-d374-4a42-8719-6c5037934faf" containerName="util" Oct 30 16:51:05.527796 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.527714 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4096055-d374-4a42-8719-6c5037934faf" containerName="util" Oct 30 16:51:05.527796 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.527734 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a4096055-d374-4a42-8719-6c5037934faf" containerName="pull" Oct 30 16:51:05.527796 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.527738 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4096055-d374-4a42-8719-6c5037934faf" containerName="pull" Oct 30 16:51:05.527796 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.527796 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4096055-d374-4a42-8719-6c5037934faf" containerName="extract" Oct 30 16:51:05.529840 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.529823 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-g226q" Oct 30 16:51:05.559512 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.559478 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-kc77p\"" Oct 30 16:51:05.560918 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.560879 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-g226q"] Oct 30 16:51:05.618670 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.618632 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68f7d8a2-f8fe-45de-9645-d0c652b338ee-bound-sa-token\") pod \"cert-manager-597b95cb84-g226q\" (UID: \"68f7d8a2-f8fe-45de-9645-d0c652b338ee\") " pod="cert-manager/cert-manager-597b95cb84-g226q" Oct 30 16:51:05.618670 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.618667 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h97m\" (UniqueName: \"kubernetes.io/projected/68f7d8a2-f8fe-45de-9645-d0c652b338ee-kube-api-access-7h97m\") pod \"cert-manager-597b95cb84-g226q\" (UID: \"68f7d8a2-f8fe-45de-9645-d0c652b338ee\") " pod="cert-manager/cert-manager-597b95cb84-g226q" Oct 30 16:51:05.719398 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.719365 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68f7d8a2-f8fe-45de-9645-d0c652b338ee-bound-sa-token\") pod \"cert-manager-597b95cb84-g226q\" (UID: \"68f7d8a2-f8fe-45de-9645-d0c652b338ee\") " pod="cert-manager/cert-manager-597b95cb84-g226q" Oct 30 16:51:05.719398 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.719402 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h97m\" (UniqueName: \"kubernetes.io/projected/68f7d8a2-f8fe-45de-9645-d0c652b338ee-kube-api-access-7h97m\") pod \"cert-manager-597b95cb84-g226q\" (UID: \"68f7d8a2-f8fe-45de-9645-d0c652b338ee\") " pod="cert-manager/cert-manager-597b95cb84-g226q" Oct 30 16:51:05.744982 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.744953 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68f7d8a2-f8fe-45de-9645-d0c652b338ee-bound-sa-token\") pod \"cert-manager-597b95cb84-g226q\" (UID: \"68f7d8a2-f8fe-45de-9645-d0c652b338ee\") " pod="cert-manager/cert-manager-597b95cb84-g226q" Oct 30 16:51:05.745184 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.745160 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h97m\" (UniqueName: \"kubernetes.io/projected/68f7d8a2-f8fe-45de-9645-d0c652b338ee-kube-api-access-7h97m\") pod \"cert-manager-597b95cb84-g226q\" (UID: \"68f7d8a2-f8fe-45de-9645-d0c652b338ee\") " pod="cert-manager/cert-manager-597b95cb84-g226q" Oct 30 16:51:05.839674 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:05.839586 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-g226q" Oct 30 16:51:06.143301 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:06.143270 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-g226q"] Oct 30 16:51:06.144462 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:51:06.144432 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68f7d8a2_f8fe_45de_9645_d0c652b338ee.slice/crio-dcfbe89a77dbccdb07d765bff6a86a52c96d971123ada52536571a96ced673fd WatchSource:0}: Error finding container dcfbe89a77dbccdb07d765bff6a86a52c96d971123ada52536571a96ced673fd: Status 404 returned error can't find the container with id dcfbe89a77dbccdb07d765bff6a86a52c96d971123ada52536571a96ced673fd Oct 30 16:51:06.519727 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:06.519695 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-g226q" event={"ID":"68f7d8a2-f8fe-45de-9645-d0c652b338ee","Type":"ContainerStarted","Data":"8fa6e79f17c762ee415a9c2e9ee7a8784120c6511474f74135df17cd276ca6c2"} Oct 30 16:51:06.519922 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:06.519734 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-g226q" event={"ID":"68f7d8a2-f8fe-45de-9645-d0c652b338ee","Type":"ContainerStarted","Data":"dcfbe89a77dbccdb07d765bff6a86a52c96d971123ada52536571a96ced673fd"} Oct 30 16:51:06.564491 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:06.564435 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-g226q" podStartSLOduration=1.564420373 podStartE2EDuration="1.564420373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:06.563391028 +0000 UTC m=+244.208901207" watchObservedRunningTime="2026-10-30 16:51:06.564420373 +0000 UTC m=+244.209930552" Oct 30 16:51:07.908917 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:51:07.908869 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:51:07.963490 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:07.963459 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:08.126403 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:08.126369 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:14.852078 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:14.852042 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:15.010544 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:15.010509 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:16.762191 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:16.762157 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:17.332471 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.332436 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw"] Oct 30 16:51:17.334772 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.334755 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" Oct 30 16:51:17.365388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.365344 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-h249r\"" Oct 30 16:51:17.365562 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.365432 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:51:17.365562 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.365499 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:51:17.366914 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.366881 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw"] Oct 30 16:51:17.404755 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.404717 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw\" (UID: \"11c8827c-b011-4db2-9a3e-542b8814113a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" Oct 30 16:51:17.404755 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.404757 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skw6l\" (UniqueName: \"kubernetes.io/projected/11c8827c-b011-4db2-9a3e-542b8814113a-kube-api-access-skw6l\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw\" (UID: \"11c8827c-b011-4db2-9a3e-542b8814113a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" Oct 30 16:51:17.405055 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.404779 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw\" (UID: \"11c8827c-b011-4db2-9a3e-542b8814113a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" Oct 30 16:51:17.505302 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.505266 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw\" (UID: \"11c8827c-b011-4db2-9a3e-542b8814113a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" Oct 30 16:51:17.505470 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.505344 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw\" (UID: \"11c8827c-b011-4db2-9a3e-542b8814113a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" Oct 30 16:51:17.505470 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.505366 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skw6l\" (UniqueName: \"kubernetes.io/projected/11c8827c-b011-4db2-9a3e-542b8814113a-kube-api-access-skw6l\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw\" (UID: \"11c8827c-b011-4db2-9a3e-542b8814113a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" Oct 30 16:51:17.505680 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.505659 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw\" (UID: \"11c8827c-b011-4db2-9a3e-542b8814113a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" Oct 30 16:51:17.505723 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.505684 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw\" (UID: \"11c8827c-b011-4db2-9a3e-542b8814113a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" Oct 30 16:51:17.530668 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.530631 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skw6l\" (UniqueName: \"kubernetes.io/projected/11c8827c-b011-4db2-9a3e-542b8814113a-kube-api-access-skw6l\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw\" (UID: \"11c8827c-b011-4db2-9a3e-542b8814113a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" Oct 30 16:51:17.648364 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.648288 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" Oct 30 16:51:17.949388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:17.949357 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw"] Oct 30 16:51:17.951799 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:51:17.951772 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c8827c_b011_4db2_9a3e_542b8814113a.slice/crio-199964f87b69736fd84af289cc938c20a856caf2bdfb14797652d3b207066824 WatchSource:0}: Error finding container 199964f87b69736fd84af289cc938c20a856caf2bdfb14797652d3b207066824: Status 404 returned error can't find the container with id 199964f87b69736fd84af289cc938c20a856caf2bdfb14797652d3b207066824 Oct 30 16:51:18.570090 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:18.570057 4828 generic.go:356] "Generic (PLEG): container finished" podID="11c8827c-b011-4db2-9a3e-542b8814113a" containerID="d2d1fcef35cc5913a54323b886859c78aeded3c9f41296d2bef05fd61f6cc968" exitCode=0 Oct 30 16:51:18.570256 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:18.570151 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" event={"ID":"11c8827c-b011-4db2-9a3e-542b8814113a","Type":"ContainerDied","Data":"d2d1fcef35cc5913a54323b886859c78aeded3c9f41296d2bef05fd61f6cc968"} Oct 30 16:51:18.570256 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:18.570211 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" event={"ID":"11c8827c-b011-4db2-9a3e-542b8814113a","Type":"ContainerStarted","Data":"199964f87b69736fd84af289cc938c20a856caf2bdfb14797652d3b207066824"} Oct 30 16:51:19.576366 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:19.576332 4828 generic.go:356] "Generic (PLEG): container finished" podID="11c8827c-b011-4db2-9a3e-542b8814113a" containerID="ff3959af9166d77dfdb78e15141453a8ed30c9dbbe7538593a82b56ae3996b85" exitCode=0 Oct 30 16:51:19.576752 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:19.576401 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" event={"ID":"11c8827c-b011-4db2-9a3e-542b8814113a","Type":"ContainerDied","Data":"ff3959af9166d77dfdb78e15141453a8ed30c9dbbe7538593a82b56ae3996b85"} Oct 30 16:51:20.582419 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:20.582381 4828 generic.go:356] "Generic (PLEG): container finished" podID="11c8827c-b011-4db2-9a3e-542b8814113a" containerID="a4cb585316202bf5447de5fb2573813051fa54dcf4e2c1170603fa4b5a12ce13" exitCode=0 Oct 30 16:51:20.582812 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:20.582432 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" event={"ID":"11c8827c-b011-4db2-9a3e-542b8814113a","Type":"ContainerDied","Data":"a4cb585316202bf5447de5fb2573813051fa54dcf4e2c1170603fa4b5a12ce13"} Oct 30 16:51:21.708561 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:21.708534 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" Oct 30 16:51:21.736429 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:21.736398 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c8827c-b011-4db2-9a3e-542b8814113a-kube-api-access-skw6l\" (UniqueName: \"kubernetes.io/projected/11c8827c-b011-4db2-9a3e-542b8814113a-kube-api-access-skw6l\") pod \"11c8827c-b011-4db2-9a3e-542b8814113a\" (UID: \"11c8827c-b011-4db2-9a3e-542b8814113a\") " Oct 30 16:51:21.736553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:21.736463 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-bundle\") pod \"11c8827c-b011-4db2-9a3e-542b8814113a\" (UID: \"11c8827c-b011-4db2-9a3e-542b8814113a\") " Oct 30 16:51:21.736553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:21.736492 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-util\" (UniqueName: \"kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-util\") pod \"11c8827c-b011-4db2-9a3e-542b8814113a\" (UID: \"11c8827c-b011-4db2-9a3e-542b8814113a\") " Oct 30 16:51:21.737063 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:21.737033 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-bundle" pod "11c8827c-b011-4db2-9a3e-542b8814113a" (UID: "11c8827c-b011-4db2-9a3e-542b8814113a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:21.738465 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:21.738441 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c8827c-b011-4db2-9a3e-542b8814113a-kube-api-access-skw6l" pod "11c8827c-b011-4db2-9a3e-542b8814113a" (UID: "11c8827c-b011-4db2-9a3e-542b8814113a"). InnerVolumeSpecName "kube-api-access-skw6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:21.744628 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:21.744596 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-util" pod "11c8827c-b011-4db2-9a3e-542b8814113a" (UID: "11c8827c-b011-4db2-9a3e-542b8814113a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:21.837265 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:21.837118 4828 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-bundle\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:51:21.837265 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:21.837147 4828 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11c8827c-b011-4db2-9a3e-542b8814113a-util\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:51:21.837265 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:21.837156 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skw6l\" (UniqueName: \"kubernetes.io/projected/11c8827c-b011-4db2-9a3e-542b8814113a-kube-api-access-skw6l\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:51:22.592803 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:22.592759 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" event={"ID":"11c8827c-b011-4db2-9a3e-542b8814113a","Type":"ContainerDied","Data":"199964f87b69736fd84af289cc938c20a856caf2bdfb14797652d3b207066824"} Oct 30 16:51:22.592803 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:22.592798 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g9rxw" Oct 30 16:51:22.592803 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:22.592810 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="199964f87b69736fd84af289cc938c20a856caf2bdfb14797652d3b207066824" Oct 30 16:51:27.350067 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:27.350022 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:28.135348 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.135313 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj"] Oct 30 16:51:28.135625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.135613 4828 cpu_manager.go:414] "containerMap: removing container" podUID="11c8827c-b011-4db2-9a3e-542b8814113a" containerName="pull" Oct 30 16:51:28.135673 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.135626 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c8827c-b011-4db2-9a3e-542b8814113a" containerName="pull" Oct 30 16:51:28.135673 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.135645 4828 cpu_manager.go:414] "containerMap: removing container" podUID="11c8827c-b011-4db2-9a3e-542b8814113a" containerName="util" Oct 30 16:51:28.135673 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.135650 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c8827c-b011-4db2-9a3e-542b8814113a" containerName="util" Oct 30 16:51:28.135673 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.135668 4828 cpu_manager.go:414] "containerMap: removing container" podUID="11c8827c-b011-4db2-9a3e-542b8814113a" containerName="extract" Oct 30 16:51:28.135673 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.135673 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c8827c-b011-4db2-9a3e-542b8814113a" containerName="extract" Oct 30 16:51:28.135839 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.135729 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c8827c-b011-4db2-9a3e-542b8814113a" containerName="extract" Oct 30 16:51:28.138161 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.138129 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.164196 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.164165 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 16:51:28.167538 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.167524 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vn468\"" Oct 30 16:51:28.167618 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.167600 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 16:51:28.167842 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.167829 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 16:51:28.167982 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.167969 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 16:51:28.186495 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.186460 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c52cbadc-73a6-4a8c-976d-9d118208b84d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.186671 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.186502 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpq5x\" (UniqueName: \"kubernetes.io/projected/c52cbadc-73a6-4a8c-976d-9d118208b84d-kube-api-access-jpq5x\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.186671 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.186532 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c52cbadc-73a6-4a8c-976d-9d118208b84d-metrics-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.186671 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.186659 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c52cbadc-73a6-4a8c-976d-9d118208b84d-tmp\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.186810 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.186713 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c52cbadc-73a6-4a8c-976d-9d118208b84d-manifests\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.186810 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.186795 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c52cbadc-73a6-4a8c-976d-9d118208b84d-webhook-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.189792 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.189764 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj"] Oct 30 16:51:28.270672 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.270635 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 16:51:28.288176 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.288144 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c52cbadc-73a6-4a8c-976d-9d118208b84d-tmp\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.288331 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.288201 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c52cbadc-73a6-4a8c-976d-9d118208b84d-manifests\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.288331 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.288231 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c52cbadc-73a6-4a8c-976d-9d118208b84d-webhook-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.288331 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.288246 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c52cbadc-73a6-4a8c-976d-9d118208b84d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.288331 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.288269 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpq5x\" (UniqueName: \"kubernetes.io/projected/c52cbadc-73a6-4a8c-976d-9d118208b84d-kube-api-access-jpq5x\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.288331 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.288297 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c52cbadc-73a6-4a8c-976d-9d118208b84d-metrics-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.288727 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.288696 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c52cbadc-73a6-4a8c-976d-9d118208b84d-manifests\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.290717 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.290690 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c52cbadc-73a6-4a8c-976d-9d118208b84d-metrics-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.290832 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.290798 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c52cbadc-73a6-4a8c-976d-9d118208b84d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.290876 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.290847 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c52cbadc-73a6-4a8c-976d-9d118208b84d-tmp\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.291070 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.291054 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c52cbadc-73a6-4a8c-976d-9d118208b84d-webhook-cert\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.323637 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.323606 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpq5x\" (UniqueName: \"kubernetes.io/projected/c52cbadc-73a6-4a8c-976d-9d118208b84d-kube-api-access-jpq5x\") pod \"opendatahub-operator-controller-manager-68c46f97f5-k89lj\" (UID: \"c52cbadc-73a6-4a8c-976d-9d118208b84d\") " pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.393160 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.393077 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:28.451302 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.451269 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:28.551790 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.551756 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:28.703759 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.703600 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:28.809410 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:28.809373 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj"] Oct 30 16:51:28.811265 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:51:28.811237 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52cbadc_73a6_4a8c_976d_9d118208b84d.slice/crio-61d074f96810355bc4f26d41c8e09bed23b18fc7714d61157d81f1f74ad4ae03 WatchSource:0}: Error finding container 61d074f96810355bc4f26d41c8e09bed23b18fc7714d61157d81f1f74ad4ae03: Status 404 returned error can't find the container with id 61d074f96810355bc4f26d41c8e09bed23b18fc7714d61157d81f1f74ad4ae03 Oct 30 16:51:29.622509 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:29.622473 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" event={"ID":"c52cbadc-73a6-4a8c-976d-9d118208b84d","Type":"ContainerStarted","Data":"61d074f96810355bc4f26d41c8e09bed23b18fc7714d61157d81f1f74ad4ae03"} Oct 30 16:51:31.221228 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:31.221194 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:31.399516 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:31.399479 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:31.696595 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:31.696564 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:31.856816 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:31.856775 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:32.636220 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:32.636180 4828 generic.go:356] "Generic (PLEG): container finished" podID="c52cbadc-73a6-4a8c-976d-9d118208b84d" containerID="31aebd382b4083c3be7aca6f7fa78ecbe6049b8884a2460e6dc256441178efbc" exitCode=0 Oct 30 16:51:32.636646 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:32.636274 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" event={"ID":"c52cbadc-73a6-4a8c-976d-9d118208b84d","Type":"ContainerDied","Data":"31aebd382b4083c3be7aca6f7fa78ecbe6049b8884a2460e6dc256441178efbc"} Oct 30 16:51:33.635789 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:33.635745 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:33.641785 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:33.641754 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" event={"ID":"c52cbadc-73a6-4a8c-976d-9d118208b84d","Type":"ContainerStarted","Data":"b3748071a74f1f928a2fe95801d895c4a0369137c4d4466d5cfdd88b05910c16"} Oct 30 16:51:33.642180 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:33.641949 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:33.737939 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:33.737861 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" podStartSLOduration=2.06901464 podStartE2EDuration="5.737841914s" totalImagesPullingTime="3.183565041s" totalInitContainerRuntime="485.262233ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:33.723710143 +0000 UTC m=+271.369220322" watchObservedRunningTime="2026-10-30 16:51:33.737841914 +0000 UTC m=+271.383352093" Oct 30 16:51:33.796047 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:33.796010 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:36.524323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:36.524278 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:36.671182 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:36.671149 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:37.908685 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:51:37.908649 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:51:44.648674 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:44.648646 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68c46f97f5-k89lj" Oct 30 16:51:47.791937 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:47.791884 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:48.058071 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.057976 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:48.294373 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.294338 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4"] Oct 30 16:51:48.296778 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.296756 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" Oct 30 16:51:48.323360 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.323293 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4"] Oct 30 16:51:48.325854 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.325828 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:48.356995 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.356963 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebe8bb77-c61f-4429-a912-26e22cea3c02-tmp\") pod \"kube-auth-proxy-768ffddf88-mg4t4\" (UID: \"ebe8bb77-c61f-4429-a912-26e22cea3c02\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" Oct 30 16:51:48.357144 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.357015 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zd6l\" (UniqueName: \"kubernetes.io/projected/ebe8bb77-c61f-4429-a912-26e22cea3c02-kube-api-access-4zd6l\") pod \"kube-auth-proxy-768ffddf88-mg4t4\" (UID: \"ebe8bb77-c61f-4429-a912-26e22cea3c02\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" Oct 30 16:51:48.357144 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.357108 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebe8bb77-c61f-4429-a912-26e22cea3c02-tls-certs\") pod \"kube-auth-proxy-768ffddf88-mg4t4\" (UID: \"ebe8bb77-c61f-4429-a912-26e22cea3c02\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" Oct 30 16:51:48.420534 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.420498 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 16:51:48.420534 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.420511 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 16:51:48.420743 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.420604 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bg78w\"" Oct 30 16:51:48.458375 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.458344 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebe8bb77-c61f-4429-a912-26e22cea3c02-tmp\") pod \"kube-auth-proxy-768ffddf88-mg4t4\" (UID: \"ebe8bb77-c61f-4429-a912-26e22cea3c02\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" Oct 30 16:51:48.458526 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.458388 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zd6l\" (UniqueName: \"kubernetes.io/projected/ebe8bb77-c61f-4429-a912-26e22cea3c02-kube-api-access-4zd6l\") pod \"kube-auth-proxy-768ffddf88-mg4t4\" (UID: \"ebe8bb77-c61f-4429-a912-26e22cea3c02\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" Oct 30 16:51:48.458526 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.458428 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebe8bb77-c61f-4429-a912-26e22cea3c02-tls-certs\") pod \"kube-auth-proxy-768ffddf88-mg4t4\" (UID: \"ebe8bb77-c61f-4429-a912-26e22cea3c02\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" Oct 30 16:51:48.460551 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.460525 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebe8bb77-c61f-4429-a912-26e22cea3c02-tmp\") pod \"kube-auth-proxy-768ffddf88-mg4t4\" (UID: \"ebe8bb77-c61f-4429-a912-26e22cea3c02\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" Oct 30 16:51:48.460803 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.460783 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebe8bb77-c61f-4429-a912-26e22cea3c02-tls-certs\") pod \"kube-auth-proxy-768ffddf88-mg4t4\" (UID: \"ebe8bb77-c61f-4429-a912-26e22cea3c02\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" Oct 30 16:51:48.493793 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.493763 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zd6l\" (UniqueName: \"kubernetes.io/projected/ebe8bb77-c61f-4429-a912-26e22cea3c02-kube-api-access-4zd6l\") pod \"kube-auth-proxy-768ffddf88-mg4t4\" (UID: \"ebe8bb77-c61f-4429-a912-26e22cea3c02\") " pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" Oct 30 16:51:48.518826 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.518793 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:48.607822 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.607744 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" Oct 30 16:51:48.912197 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:48.912171 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4"] Oct 30 16:51:48.914082 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:51:48.914053 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe8bb77_c61f_4429_a912_26e22cea3c02.slice/crio-e571588c6d7f435d07bdd608fb15ed8d031094e44b8d7489a4223cdad235d55d WatchSource:0}: Error finding container e571588c6d7f435d07bdd608fb15ed8d031094e44b8d7489a4223cdad235d55d: Status 404 returned error can't find the container with id e571588c6d7f435d07bdd608fb15ed8d031094e44b8d7489a4223cdad235d55d Oct 30 16:51:49.711737 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:49.711659 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" event={"ID":"ebe8bb77-c61f-4429-a912-26e22cea3c02","Type":"ContainerStarted","Data":"e571588c6d7f435d07bdd608fb15ed8d031094e44b8d7489a4223cdad235d55d"} Oct 30 16:51:50.034610 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.034575 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk"] Oct 30 16:51:50.036818 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.036800 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.052862 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.052828 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 16:51:50.053046 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.052828 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 16:51:50.053046 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.052828 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 16:51:50.053046 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.052828 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 16:51:50.053046 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.052976 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xjmfv\"" Oct 30 16:51:50.068448 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.068414 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk"] Oct 30 16:51:50.100721 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.100687 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:50.176276 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.176228 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.176449 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.176293 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.176449 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.176368 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.176449 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.176423 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.176577 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.176551 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.176618 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.176600 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d1b9f2-99c0-413d-833b-775b2a895879-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.176695 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.176680 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6jj4\" (UniqueName: \"kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-kube-api-access-p6jj4\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.269806 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.269700 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:50.277811 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.277772 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6jj4\" (UniqueName: \"kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-kube-api-access-p6jj4\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.277999 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.277813 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.277999 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.277855 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.277999 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.277883 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.277999 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.277957 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.278224 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.278013 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.278224 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.278041 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d1b9f2-99c0-413d-833b-775b2a895879-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.278714 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.278656 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.280883 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.280855 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d1b9f2-99c0-413d-833b-775b2a895879-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.281079 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.281052 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.281170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.281058 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.282218 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.282191 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.302726 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.302652 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.302891 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.302865 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6jj4\" (UniqueName: \"kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-kube-api-access-p6jj4\") pod \"istiod-openshift-gateway-7876bfcb74-6xxsk\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.348008 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.347971 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:50.664962 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.664927 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk"] Oct 30 16:51:50.666315 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:51:50.666285 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d1b9f2_99c0_413d_833b_775b2a895879.slice/crio-b4545a9a5ab4ab7ce088cbae16a0d4d7090ddb4fd989f34ad4ba5634ca7394f7 WatchSource:0}: Error finding container b4545a9a5ab4ab7ce088cbae16a0d4d7090ddb4fd989f34ad4ba5634ca7394f7: Status 404 returned error can't find the container with id b4545a9a5ab4ab7ce088cbae16a0d4d7090ddb4fd989f34ad4ba5634ca7394f7 Oct 30 16:51:50.717254 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:50.717210 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" event={"ID":"37d1b9f2-99c0-413d-833b-775b2a895879","Type":"ContainerStarted","Data":"b4545a9a5ab4ab7ce088cbae16a0d4d7090ddb4fd989f34ad4ba5634ca7394f7"} Oct 30 16:51:51.563478 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:51.563424 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:51.727920 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:51.722895 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:51.730616 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:51.730584 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" event={"ID":"ebe8bb77-c61f-4429-a912-26e22cea3c02","Type":"ContainerStarted","Data":"c2c3a94af8203c2e7439e8f52d15120d021017966a1fb62eb419ad0fc3cc1c49"} Oct 30 16:51:51.778044 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:51.777985 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-768ffddf88-mg4t4" podStartSLOduration=1.561931094 podStartE2EDuration="3.777970944s" totalImagesPullingTime="2.21603985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:51.777285083 +0000 UTC m=+289.422795263" watchObservedRunningTime="2026-10-30 16:51:51.777970944 +0000 UTC m=+289.423481186" Oct 30 16:51:51.935811 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:51.935725 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:52.450363 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:52.450329 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:52.844974 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:52.844936 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:53.198655 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:53.198624 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:51:53.198750 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:53.198707 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:51:53.742843 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:53.742800 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" event={"ID":"37d1b9f2-99c0-413d-833b-775b2a895879","Type":"ContainerStarted","Data":"32d599b63a9cdec14cb9506ee20a3ee0f52843cdc0d7429f243654169c59b6f9"} Oct 30 16:51:53.743042 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:53.742934 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:53.824409 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:53.824355 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" podStartSLOduration=1.295229014 podStartE2EDuration="3.824340811s" totalImagesPullingTime="2.529111797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:53.809644705 +0000 UTC m=+291.455154907" watchObservedRunningTime="2026-10-30 16:51:53.824340811 +0000 UTC m=+291.469850994" Oct 30 16:51:54.167128 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:54.167089 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:54.315760 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:54.315720 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:54.591745 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:54.591692 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:54.749581 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:54.749553 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:51:54.760055 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:54.759876 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:51:54.957239 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:51:54.957155 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:02.502774 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:02.502745 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 16:52:02.503301 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:02.503252 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 16:52:05.827469 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:05.827435 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:05.858755 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:05.858715 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v"] Oct 30 16:52:05.898186 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:05.898153 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v"] Oct 30 16:52:05.898360 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:05.898300 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:05.914987 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:05.914961 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-zhg82\"" Oct 30 16:52:05.943010 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:05.942977 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 16:52:06.015331 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.015290 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7x5\" (UniqueName: \"kubernetes.io/projected/1f68ded2-8cca-48d6-bbba-84e747e3b6cb-kube-api-access-4j7x5\") pod \"kserve-module-controller-manager-5cfb77bf8c-rgk8v\" (UID: \"1f68ded2-8cca-48d6-bbba-84e747e3b6cb\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:06.015536 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.015362 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f68ded2-8cca-48d6-bbba-84e747e3b6cb-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-rgk8v\" (UID: \"1f68ded2-8cca-48d6-bbba-84e747e3b6cb\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:06.015536 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.015447 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f68ded2-8cca-48d6-bbba-84e747e3b6cb-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-rgk8v\" (UID: \"1f68ded2-8cca-48d6-bbba-84e747e3b6cb\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:06.015536 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.015502 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1f68ded2-8cca-48d6-bbba-84e747e3b6cb-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-rgk8v\" (UID: \"1f68ded2-8cca-48d6-bbba-84e747e3b6cb\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:06.026310 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.026279 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:06.116031 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.115933 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f68ded2-8cca-48d6-bbba-84e747e3b6cb-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-rgk8v\" (UID: \"1f68ded2-8cca-48d6-bbba-84e747e3b6cb\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:06.116031 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.115983 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f68ded2-8cca-48d6-bbba-84e747e3b6cb-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-rgk8v\" (UID: \"1f68ded2-8cca-48d6-bbba-84e747e3b6cb\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:06.116031 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.116015 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1f68ded2-8cca-48d6-bbba-84e747e3b6cb-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-rgk8v\" (UID: \"1f68ded2-8cca-48d6-bbba-84e747e3b6cb\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:06.116300 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.116064 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7x5\" (UniqueName: \"kubernetes.io/projected/1f68ded2-8cca-48d6-bbba-84e747e3b6cb-kube-api-access-4j7x5\") pod \"kserve-module-controller-manager-5cfb77bf8c-rgk8v\" (UID: \"1f68ded2-8cca-48d6-bbba-84e747e3b6cb\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:06.116365 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.116347 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f68ded2-8cca-48d6-bbba-84e747e3b6cb-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-rgk8v\" (UID: \"1f68ded2-8cca-48d6-bbba-84e747e3b6cb\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:06.116412 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.116380 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1f68ded2-8cca-48d6-bbba-84e747e3b6cb-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-rgk8v\" (UID: \"1f68ded2-8cca-48d6-bbba-84e747e3b6cb\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:06.116957 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.116940 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f68ded2-8cca-48d6-bbba-84e747e3b6cb-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-rgk8v\" (UID: \"1f68ded2-8cca-48d6-bbba-84e747e3b6cb\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:06.141401 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.141370 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7x5\" (UniqueName: \"kubernetes.io/projected/1f68ded2-8cca-48d6-bbba-84e747e3b6cb-kube-api-access-4j7x5\") pod \"kserve-module-controller-manager-5cfb77bf8c-rgk8v\" (UID: \"1f68ded2-8cca-48d6-bbba-84e747e3b6cb\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:06.211440 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.211401 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:06.548483 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.548455 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v"] Oct 30 16:52:06.549829 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:52:06.549801 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f68ded2_8cca_48d6_bbba_84e747e3b6cb.slice/crio-1e5041a5bb1cb848992b9c15211dc7c646327ed9abcd89c81c4227376b4f6c99 WatchSource:0}: Error finding container 1e5041a5bb1cb848992b9c15211dc7c646327ed9abcd89c81c4227376b4f6c99: Status 404 returned error can't find the container with id 1e5041a5bb1cb848992b9c15211dc7c646327ed9abcd89c81c4227376b4f6c99 Oct 30 16:52:06.552008 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.551990 4828 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:52:06.806174 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:06.806089 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" event={"ID":"1f68ded2-8cca-48d6-bbba-84e747e3b6cb","Type":"ContainerStarted","Data":"1e5041a5bb1cb848992b9c15211dc7c646327ed9abcd89c81c4227376b4f6c99"} Oct 30 16:52:07.920087 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:52:07.920045 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:52:08.816168 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:08.816125 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" event={"ID":"1f68ded2-8cca-48d6-bbba-84e747e3b6cb","Type":"ContainerStarted","Data":"1bd13d70954081e63b32255d9ea1957b1fc5c68583462b776dcf6d278f014426"} Oct 30 16:52:08.816450 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:08.816206 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:08.879002 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:08.878944 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" podStartSLOduration=2.429535056 podStartE2EDuration="3.878932296s" totalImagesPullingTime="1.44939724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:08.87848576 +0000 UTC m=+306.523995941" watchObservedRunningTime="2026-10-30 16:52:08.878932296 +0000 UTC m=+306.524442476" Oct 30 16:52:29.883847 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:29.883754 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-rgk8v" Oct 30 16:52:37.909341 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:52:37.909294 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:52:42.967097 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:42.967053 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:43.137178 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:43.137138 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:52.332242 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.332204 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9"] Oct 30 16:52:52.336471 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.336444 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.393754 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.393699 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32fbfdf-cbf9-4cf3-a261-bce740046772-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.393754 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.393757 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.394008 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.393843 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.394008 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.393978 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.394123 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.394038 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.394123 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.394087 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2gp\" (UniqueName: \"kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-kube-api-access-jh2gp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.394225 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.394149 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.464471 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.464434 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9"] Oct 30 16:52:52.495316 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.495277 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.495480 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.495343 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.495480 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.495380 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.495480 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.495404 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2gp\" (UniqueName: \"kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-kube-api-access-jh2gp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.495480 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.495422 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.495480 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.495439 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32fbfdf-cbf9-4cf3-a261-bce740046772-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.495480 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.495470 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.496092 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.496061 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.497944 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.497890 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32fbfdf-cbf9-4cf3-a261-bce740046772-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.498041 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.497947 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.498121 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.498097 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.498157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.498129 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.520458 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.520433 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.520771 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.520749 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2gp\" (UniqueName: \"kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-kube-api-access-jh2gp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xxzj9\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.647927 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.647819 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:52.967403 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.967370 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9"] Oct 30 16:52:52.969393 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:52:52.969364 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32fbfdf_cbf9_4cf3_a261_bce740046772.slice/crio-85a98911b4f9e6dd8c558ce73818969d4f1f94798a74bf4e7b9cc7abe7277c1b WatchSource:0}: Error finding container 85a98911b4f9e6dd8c558ce73818969d4f1f94798a74bf4e7b9cc7abe7277c1b: Status 404 returned error can't find the container with id 85a98911b4f9e6dd8c558ce73818969d4f1f94798a74bf4e7b9cc7abe7277c1b Oct 30 16:52:52.971265 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.971228 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:52:52.971364 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.971292 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:52:52.992215 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:52.992189 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:53.024895 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:53.024862 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" event={"ID":"b32fbfdf-cbf9-4cf3-a261-bce740046772","Type":"ContainerStarted","Data":"85a98911b4f9e6dd8c558ce73818969d4f1f94798a74bf4e7b9cc7abe7277c1b"} Oct 30 16:52:53.319572 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:53.319527 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:53.496006 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:53.495966 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:53.659940 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:53.659822 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:53.815602 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:53.815565 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:54.031520 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.031482 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" event={"ID":"b32fbfdf-cbf9-4cf3-a261-bce740046772","Type":"ContainerStarted","Data":"99c3ca732312c8785f696e01bc56fc3827f09f2797e98f65e758b381ef31258d"} Oct 30 16:52:54.031727 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.031675 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:54.033319 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.033299 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 16:52:54.086784 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.086748 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:54.108437 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.108384 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" podStartSLOduration=2.108370434 podStartE2EDuration="2.108370434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:54.093579572 +0000 UTC m=+351.739089754" watchObservedRunningTime="2026-10-30 16:52:54.108370434 +0000 UTC m=+351.753880614" Oct 30 16:52:54.179264 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.179232 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk"] Oct 30 16:52:54.179498 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.179466 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" podUID="37d1b9f2-99c0-413d-833b-775b2a895879" containerName="discovery" containerID="cri-o://32d599b63a9cdec14cb9506ee20a3ee0f52843cdc0d7429f243654169c59b6f9" gracePeriod=30 Oct 30 16:52:54.418814 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.418787 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:52:54.512181 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.512153 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-kubeconfig\") pod \"37d1b9f2-99c0-413d-833b-775b2a895879\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " Oct 30 16:52:54.512657 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.512200 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d1b9f2-99c0-413d-833b-775b2a895879-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d1b9f2-99c0-413d-833b-775b2a895879-local-certs\") pod \"37d1b9f2-99c0-413d-833b-775b2a895879\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " Oct 30 16:52:54.512657 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.512222 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-kube-api-access-p6jj4\" (UniqueName: \"kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-kube-api-access-p6jj4\") pod \"37d1b9f2-99c0-413d-833b-775b2a895879\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " Oct 30 16:52:54.512657 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.512263 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-ca-configmap\") pod \"37d1b9f2-99c0-413d-833b-775b2a895879\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " Oct 30 16:52:54.512657 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.512322 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-istio-token\" (UniqueName: \"kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-istio-token\") pod \"37d1b9f2-99c0-413d-833b-775b2a895879\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " Oct 30 16:52:54.512657 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.512380 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-dns-cert\") pod \"37d1b9f2-99c0-413d-833b-775b2a895879\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " Oct 30 16:52:54.512657 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.512416 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-cacerts\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-cacerts\") pod \"37d1b9f2-99c0-413d-833b-775b2a895879\" (UID: \"37d1b9f2-99c0-413d-833b-775b2a895879\") " Oct 30 16:52:54.513017 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.512727 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-ca-configmap" pod "37d1b9f2-99c0-413d-833b-775b2a895879" (UID: "37d1b9f2-99c0-413d-833b-775b2a895879"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:54.515049 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.515017 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-kubeconfig" pod "37d1b9f2-99c0-413d-833b-775b2a895879" (UID: "37d1b9f2-99c0-413d-833b-775b2a895879"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:54.515171 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.515038 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d1b9f2-99c0-413d-833b-775b2a895879-local-certs" pod "37d1b9f2-99c0-413d-833b-775b2a895879" (UID: "37d1b9f2-99c0-413d-833b-775b2a895879"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:54.515171 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.515111 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-dns-cert" pod "37d1b9f2-99c0-413d-833b-775b2a895879" (UID: "37d1b9f2-99c0-413d-833b-775b2a895879"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:54.515279 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.515163 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-kube-api-access-p6jj4" pod "37d1b9f2-99c0-413d-833b-775b2a895879" (UID: "37d1b9f2-99c0-413d-833b-775b2a895879"). InnerVolumeSpecName "kube-api-access-p6jj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:54.515279 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.515223 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-cacerts" pod "37d1b9f2-99c0-413d-833b-775b2a895879" (UID: "37d1b9f2-99c0-413d-833b-775b2a895879"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:54.515279 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.515266 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-istio-token" pod "37d1b9f2-99c0-413d-833b-775b2a895879" (UID: "37d1b9f2-99c0-413d-833b-775b2a895879"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:54.613194 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.613157 4828 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-dns-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:52:54.613194 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.613189 4828 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-cacerts\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:52:54.613194 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.613199 4828 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/37d1b9f2-99c0-413d-833b-775b2a895879-istio-kubeconfig\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:52:54.613427 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.613209 4828 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d1b9f2-99c0-413d-833b-775b2a895879-local-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:52:54.613427 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.613218 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6jj4\" (UniqueName: \"kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-kube-api-access-p6jj4\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:52:54.613427 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.613228 4828 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/37d1b9f2-99c0-413d-833b-775b2a895879-istio-csr-ca-configmap\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:52:54.613427 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:54.613237 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d1b9f2-99c0-413d-833b-775b2a895879-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:52:55.037319 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:55.037286 4828 generic.go:356] "Generic (PLEG): container finished" podID="37d1b9f2-99c0-413d-833b-775b2a895879" containerID="32d599b63a9cdec14cb9506ee20a3ee0f52843cdc0d7429f243654169c59b6f9" exitCode=0 Oct 30 16:52:55.037489 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:55.037337 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" Oct 30 16:52:55.037489 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:55.037371 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" event={"ID":"37d1b9f2-99c0-413d-833b-775b2a895879","Type":"ContainerDied","Data":"32d599b63a9cdec14cb9506ee20a3ee0f52843cdc0d7429f243654169c59b6f9"} Oct 30 16:52:55.037489 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:55.037426 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk" event={"ID":"37d1b9f2-99c0-413d-833b-775b2a895879","Type":"ContainerDied","Data":"b4545a9a5ab4ab7ce088cbae16a0d4d7090ddb4fd989f34ad4ba5634ca7394f7"} Oct 30 16:52:55.037489 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:55.037447 4828 scope.go:122] "RemoveContainer" containerID="32d599b63a9cdec14cb9506ee20a3ee0f52843cdc0d7429f243654169c59b6f9" Oct 30 16:52:55.046008 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:55.045985 4828 scope.go:122] "RemoveContainer" containerID="32d599b63a9cdec14cb9506ee20a3ee0f52843cdc0d7429f243654169c59b6f9" Oct 30 16:52:55.046267 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:52:55.046250 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d599b63a9cdec14cb9506ee20a3ee0f52843cdc0d7429f243654169c59b6f9\": container with ID starting with 32d599b63a9cdec14cb9506ee20a3ee0f52843cdc0d7429f243654169c59b6f9 not found: ID does not exist" containerID="32d599b63a9cdec14cb9506ee20a3ee0f52843cdc0d7429f243654169c59b6f9" Oct 30 16:52:55.046333 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:55.046275 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d599b63a9cdec14cb9506ee20a3ee0f52843cdc0d7429f243654169c59b6f9"} err="failed to get container status \"32d599b63a9cdec14cb9506ee20a3ee0f52843cdc0d7429f243654169c59b6f9\": rpc error: code = NotFound desc = could not find container \"32d599b63a9cdec14cb9506ee20a3ee0f52843cdc0d7429f243654169c59b6f9\": container with ID starting with 32d599b63a9cdec14cb9506ee20a3ee0f52843cdc0d7429f243654169c59b6f9 not found: ID does not exist" Oct 30 16:52:55.093492 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:55.093458 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk"] Oct 30 16:52:55.106828 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:55.106798 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6xxsk"] Oct 30 16:52:55.930232 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:55.930193 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:56.081329 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:56.081287 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:56.236329 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:56.236239 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:56.389646 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:56.389605 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:52:56.487572 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:56.487480 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d1b9f2-99c0-413d-833b-775b2a895879" path="/var/lib/kubelet/pods/37d1b9f2-99c0-413d-833b-775b2a895879/volumes" Oct 30 16:52:57.334219 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:52:57.334183 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:07.909999 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:53:07.909957 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:53:37.908592 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:53:37.908555 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:53:41.333115 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:41.333076 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:41.511226 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:41.511194 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:44.103814 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.103780 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb"] Oct 30 16:53:44.104327 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.104128 4828 cpu_manager.go:414] "containerMap: removing container" podUID="37d1b9f2-99c0-413d-833b-775b2a895879" containerName="discovery" Oct 30 16:53:44.104327 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.104140 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d1b9f2-99c0-413d-833b-775b2a895879" containerName="discovery" Oct 30 16:53:44.104327 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.104213 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d1b9f2-99c0-413d-833b-775b2a895879" containerName="discovery" Oct 30 16:53:44.107539 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.107517 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" Oct 30 16:53:44.123550 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.123529 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-8lvsl\"" Oct 30 16:53:44.136966 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.136933 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb\" (UID: \"5116f303-bd76-4921-bc75-1b8573200f6c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" Oct 30 16:53:44.137096 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.136978 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8p7m\" (UniqueName: \"kubernetes.io/projected/5116f303-bd76-4921-bc75-1b8573200f6c-kube-api-access-n8p7m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb\" (UID: \"5116f303-bd76-4921-bc75-1b8573200f6c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" Oct 30 16:53:44.137096 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.137071 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb\" (UID: \"5116f303-bd76-4921-bc75-1b8573200f6c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" Oct 30 16:53:44.137252 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.137234 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:53:44.137354 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.137341 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:53:44.139017 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.138997 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb"] Oct 30 16:53:44.202178 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.202144 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr"] Oct 30 16:53:44.205677 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.205658 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" Oct 30 16:53:44.226561 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.226495 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-z2png\"" Oct 30 16:53:44.228589 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.228554 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr"] Oct 30 16:53:44.238095 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.238073 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb\" (UID: \"5116f303-bd76-4921-bc75-1b8573200f6c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" Oct 30 16:53:44.238225 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.238111 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8p7m\" (UniqueName: \"kubernetes.io/projected/5116f303-bd76-4921-bc75-1b8573200f6c-kube-api-access-n8p7m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb\" (UID: \"5116f303-bd76-4921-bc75-1b8573200f6c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" Oct 30 16:53:44.238225 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.238134 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr\" (UID: \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" Oct 30 16:53:44.238225 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.238158 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxd7h\" (UniqueName: \"kubernetes.io/projected/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-kube-api-access-rxd7h\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr\" (UID: \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" Oct 30 16:53:44.238225 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.238181 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr\" (UID: \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" Oct 30 16:53:44.238434 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.238396 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb\" (UID: \"5116f303-bd76-4921-bc75-1b8573200f6c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" Oct 30 16:53:44.238514 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.238496 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb\" (UID: \"5116f303-bd76-4921-bc75-1b8573200f6c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" Oct 30 16:53:44.238692 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.238676 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb\" (UID: \"5116f303-bd76-4921-bc75-1b8573200f6c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" Oct 30 16:53:44.263670 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.263639 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8p7m\" (UniqueName: \"kubernetes.io/projected/5116f303-bd76-4921-bc75-1b8573200f6c-kube-api-access-n8p7m\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb\" (UID: \"5116f303-bd76-4921-bc75-1b8573200f6c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" Oct 30 16:53:44.297185 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.297154 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n"] Oct 30 16:53:44.302539 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.302517 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" Oct 30 16:53:44.321892 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.321863 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-shm2k\"" Oct 30 16:53:44.323652 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.323626 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n"] Oct 30 16:53:44.339186 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.339153 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n\" (UID: \"0f16a076-c310-4aac-a65e-58ed25f29195\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" Oct 30 16:53:44.339338 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.339224 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr\" (UID: \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" Oct 30 16:53:44.339338 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.339289 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxd7h\" (UniqueName: \"kubernetes.io/projected/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-kube-api-access-rxd7h\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr\" (UID: \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" Oct 30 16:53:44.339338 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.339321 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr\" (UID: \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" Oct 30 16:53:44.339488 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.339366 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n\" (UID: \"0f16a076-c310-4aac-a65e-58ed25f29195\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" Oct 30 16:53:44.339488 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.339406 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prbps\" (UniqueName: \"kubernetes.io/projected/0f16a076-c310-4aac-a65e-58ed25f29195-kube-api-access-prbps\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n\" (UID: \"0f16a076-c310-4aac-a65e-58ed25f29195\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" Oct 30 16:53:44.339634 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.339616 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr\" (UID: \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" Oct 30 16:53:44.339681 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.339649 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr\" (UID: \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" Oct 30 16:53:44.364384 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.364317 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxd7h\" (UniqueName: \"kubernetes.io/projected/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-kube-api-access-rxd7h\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr\" (UID: \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" Oct 30 16:53:44.397509 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.397473 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s"] Oct 30 16:53:44.401397 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.401378 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" Oct 30 16:53:44.422885 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.422859 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" Oct 30 16:53:44.423355 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.423333 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-rtkrh\"" Oct 30 16:53:44.425197 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.425167 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s"] Oct 30 16:53:44.440315 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.440286 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n\" (UID: \"0f16a076-c310-4aac-a65e-58ed25f29195\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" Oct 30 16:53:44.440442 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.440334 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prbps\" (UniqueName: \"kubernetes.io/projected/0f16a076-c310-4aac-a65e-58ed25f29195-kube-api-access-prbps\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n\" (UID: \"0f16a076-c310-4aac-a65e-58ed25f29195\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" Oct 30 16:53:44.440442 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.440375 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s\" (UID: \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" Oct 30 16:53:44.440442 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.440401 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n\" (UID: \"0f16a076-c310-4aac-a65e-58ed25f29195\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" Oct 30 16:53:44.440761 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.440738 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n\" (UID: \"0f16a076-c310-4aac-a65e-58ed25f29195\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" Oct 30 16:53:44.440869 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.440847 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s\" (UID: \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" Oct 30 16:53:44.441003 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.440984 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflmp\" (UniqueName: \"kubernetes.io/projected/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-kube-api-access-kflmp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s\" (UID: \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" Oct 30 16:53:44.441082 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.441034 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n\" (UID: \"0f16a076-c310-4aac-a65e-58ed25f29195\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" Oct 30 16:53:44.466146 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.466114 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prbps\" (UniqueName: \"kubernetes.io/projected/0f16a076-c310-4aac-a65e-58ed25f29195-kube-api-access-prbps\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n\" (UID: \"0f16a076-c310-4aac-a65e-58ed25f29195\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" Oct 30 16:53:44.521280 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.521245 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" Oct 30 16:53:44.541599 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.541521 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s\" (UID: \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" Oct 30 16:53:44.541732 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.541616 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s\" (UID: \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" Oct 30 16:53:44.541732 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.541695 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflmp\" (UniqueName: \"kubernetes.io/projected/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-kube-api-access-kflmp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s\" (UID: \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" Oct 30 16:53:44.541990 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.541969 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s\" (UID: \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" Oct 30 16:53:44.542066 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.542015 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s\" (UID: \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" Oct 30 16:53:44.567202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.567171 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflmp\" (UniqueName: \"kubernetes.io/projected/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-kube-api-access-kflmp\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s\" (UID: \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" Oct 30 16:53:44.618444 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.618330 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" Oct 30 16:53:44.717582 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.717543 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" Oct 30 16:53:44.731191 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.731166 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb"] Oct 30 16:53:44.731774 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:53:44.731745 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5116f303_bd76_4921_bc75_1b8573200f6c.slice/crio-bd7ca4b325220e7e371bf9f36c5cea58edb8d59b1f8a81c38dfe86dd955c7f11 WatchSource:0}: Error finding container bd7ca4b325220e7e371bf9f36c5cea58edb8d59b1f8a81c38dfe86dd955c7f11: Status 404 returned error can't find the container with id bd7ca4b325220e7e371bf9f36c5cea58edb8d59b1f8a81c38dfe86dd955c7f11 Oct 30 16:53:44.824947 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.824922 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr"] Oct 30 16:53:44.826088 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:53:44.826057 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aad9f3e_ba60_436a_a1ae_5b848d339d9e.slice/crio-b02c3538c1d15f384b314fb1edfaf2a88750a544ba3aec118b7d0d4aed9bc4cd WatchSource:0}: Error finding container b02c3538c1d15f384b314fb1edfaf2a88750a544ba3aec118b7d0d4aed9bc4cd: Status 404 returned error can't find the container with id b02c3538c1d15f384b314fb1edfaf2a88750a544ba3aec118b7d0d4aed9bc4cd Oct 30 16:53:44.917997 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:44.917970 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n"] Oct 30 16:53:44.918938 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:53:44.918885 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f16a076_c310_4aac_a65e_58ed25f29195.slice/crio-aa30b21b3ae897189b2e795bd3287a4b1c58276988e2b264e0ad934b40a194b9 WatchSource:0}: Error finding container aa30b21b3ae897189b2e795bd3287a4b1c58276988e2b264e0ad934b40a194b9: Status 404 returned error can't find the container with id aa30b21b3ae897189b2e795bd3287a4b1c58276988e2b264e0ad934b40a194b9 Oct 30 16:53:45.020721 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:45.020688 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s"] Oct 30 16:53:45.025927 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:53:45.025870 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b19b1f_2cd9_4cee_a0d4_cdc54ad04b37.slice/crio-22aadb6525c27fd9efd0b5a3a004843bc1e4233a6ef118963c83214a35680663 WatchSource:0}: Error finding container 22aadb6525c27fd9efd0b5a3a004843bc1e4233a6ef118963c83214a35680663: Status 404 returned error can't find the container with id 22aadb6525c27fd9efd0b5a3a004843bc1e4233a6ef118963c83214a35680663 Oct 30 16:53:45.270118 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:45.270081 4828 generic.go:356] "Generic (PLEG): container finished" podID="5116f303-bd76-4921-bc75-1b8573200f6c" containerID="f999881d72de3ed09bee6bc39d2c333d09e7613275442518f732a319ca339569" exitCode=0 Oct 30 16:53:45.270594 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:45.270168 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" event={"ID":"5116f303-bd76-4921-bc75-1b8573200f6c","Type":"ContainerDied","Data":"f999881d72de3ed09bee6bc39d2c333d09e7613275442518f732a319ca339569"} Oct 30 16:53:45.270594 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:45.270231 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" event={"ID":"5116f303-bd76-4921-bc75-1b8573200f6c","Type":"ContainerStarted","Data":"bd7ca4b325220e7e371bf9f36c5cea58edb8d59b1f8a81c38dfe86dd955c7f11"} Oct 30 16:53:45.271709 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:45.271682 4828 generic.go:356] "Generic (PLEG): container finished" podID="0f16a076-c310-4aac-a65e-58ed25f29195" containerID="60636303e7391532ea7b6dca9d3317547ba53fd161f0ba957b7addb2b03aba38" exitCode=0 Oct 30 16:53:45.271826 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:45.271751 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" event={"ID":"0f16a076-c310-4aac-a65e-58ed25f29195","Type":"ContainerDied","Data":"60636303e7391532ea7b6dca9d3317547ba53fd161f0ba957b7addb2b03aba38"} Oct 30 16:53:45.271826 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:45.271797 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" event={"ID":"0f16a076-c310-4aac-a65e-58ed25f29195","Type":"ContainerStarted","Data":"aa30b21b3ae897189b2e795bd3287a4b1c58276988e2b264e0ad934b40a194b9"} Oct 30 16:53:45.273584 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:45.273561 4828 generic.go:356] "Generic (PLEG): container finished" podID="12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37" containerID="552b13b7fe00d066b1b63770c5ffdab18a9921699ea739d4668d622a67cb6bd5" exitCode=0 Oct 30 16:53:45.273678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:45.273640 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" event={"ID":"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37","Type":"ContainerDied","Data":"552b13b7fe00d066b1b63770c5ffdab18a9921699ea739d4668d622a67cb6bd5"} Oct 30 16:53:45.273796 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:45.273697 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" event={"ID":"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37","Type":"ContainerStarted","Data":"22aadb6525c27fd9efd0b5a3a004843bc1e4233a6ef118963c83214a35680663"} Oct 30 16:53:45.275390 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:45.275295 4828 generic.go:356] "Generic (PLEG): container finished" podID="2aad9f3e-ba60-436a-a1ae-5b848d339d9e" containerID="86cc7530c97277262a06cf25dbcb1692fc8408f37a28589e5c8a362eb9453b4b" exitCode=0 Oct 30 16:53:45.275390 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:45.275336 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" event={"ID":"2aad9f3e-ba60-436a-a1ae-5b848d339d9e","Type":"ContainerDied","Data":"86cc7530c97277262a06cf25dbcb1692fc8408f37a28589e5c8a362eb9453b4b"} Oct 30 16:53:45.275390 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:45.275369 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" event={"ID":"2aad9f3e-ba60-436a-a1ae-5b848d339d9e","Type":"ContainerStarted","Data":"b02c3538c1d15f384b314fb1edfaf2a88750a544ba3aec118b7d0d4aed9bc4cd"} Oct 30 16:53:46.289050 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:46.289012 4828 generic.go:356] "Generic (PLEG): container finished" podID="2aad9f3e-ba60-436a-a1ae-5b848d339d9e" containerID="9b855e4f68b3c8f0dd053e3792de17335f0ca8cdd346436b11be3d1253a5e5ff" exitCode=0 Oct 30 16:53:46.289472 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:46.289107 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" event={"ID":"2aad9f3e-ba60-436a-a1ae-5b848d339d9e","Type":"ContainerDied","Data":"9b855e4f68b3c8f0dd053e3792de17335f0ca8cdd346436b11be3d1253a5e5ff"} Oct 30 16:53:46.291450 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:46.291424 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" event={"ID":"5116f303-bd76-4921-bc75-1b8573200f6c","Type":"ContainerStarted","Data":"a349d4f074fbbde1df753b3bfe2746997a751b96f9379313b37b1b5dc0d32baf"} Oct 30 16:53:46.293954 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:46.293928 4828 generic.go:356] "Generic (PLEG): container finished" podID="0f16a076-c310-4aac-a65e-58ed25f29195" containerID="cb11a244ab801e4be1afe87592636ca64e0a563b5255e274ff952d790cea245e" exitCode=0 Oct 30 16:53:46.294060 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:46.294003 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" event={"ID":"0f16a076-c310-4aac-a65e-58ed25f29195","Type":"ContainerDied","Data":"cb11a244ab801e4be1afe87592636ca64e0a563b5255e274ff952d790cea245e"} Oct 30 16:53:47.300660 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:47.300628 4828 generic.go:356] "Generic (PLEG): container finished" podID="5116f303-bd76-4921-bc75-1b8573200f6c" containerID="a349d4f074fbbde1df753b3bfe2746997a751b96f9379313b37b1b5dc0d32baf" exitCode=0 Oct 30 16:53:47.301116 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:47.300715 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" event={"ID":"5116f303-bd76-4921-bc75-1b8573200f6c","Type":"ContainerDied","Data":"a349d4f074fbbde1df753b3bfe2746997a751b96f9379313b37b1b5dc0d32baf"} Oct 30 16:53:47.302919 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:47.302883 4828 generic.go:356] "Generic (PLEG): container finished" podID="0f16a076-c310-4aac-a65e-58ed25f29195" containerID="e9bb278882683f09ab106bad49f163421345ce6234dfdf6a866fed52d787515a" exitCode=0 Oct 30 16:53:47.303010 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:47.302940 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" event={"ID":"0f16a076-c310-4aac-a65e-58ed25f29195","Type":"ContainerDied","Data":"e9bb278882683f09ab106bad49f163421345ce6234dfdf6a866fed52d787515a"} Oct 30 16:53:47.304699 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:47.304675 4828 generic.go:356] "Generic (PLEG): container finished" podID="12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37" containerID="035073f219c2c0995ba832f3c162f1fe4586cc058e10cf9f2267734ae5fb45a5" exitCode=0 Oct 30 16:53:47.304801 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:47.304754 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" event={"ID":"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37","Type":"ContainerDied","Data":"035073f219c2c0995ba832f3c162f1fe4586cc058e10cf9f2267734ae5fb45a5"} Oct 30 16:53:47.306774 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:47.306752 4828 generic.go:356] "Generic (PLEG): container finished" podID="2aad9f3e-ba60-436a-a1ae-5b848d339d9e" containerID="a65b24c93af30a730a8f69f4ae6c96a14709b3a1d1cab136cca4d3e130af2e1d" exitCode=0 Oct 30 16:53:47.306877 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:47.306802 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" event={"ID":"2aad9f3e-ba60-436a-a1ae-5b848d339d9e","Type":"ContainerDied","Data":"a65b24c93af30a730a8f69f4ae6c96a14709b3a1d1cab136cca4d3e130af2e1d"} Oct 30 16:53:48.313678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.313632 4828 generic.go:356] "Generic (PLEG): container finished" podID="12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37" containerID="98653c9d5082aa3de89d9e96acb27803b90c43c9c6f1ec84a89f27de4b357512" exitCode=0 Oct 30 16:53:48.314094 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.313688 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" event={"ID":"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37","Type":"ContainerDied","Data":"98653c9d5082aa3de89d9e96acb27803b90c43c9c6f1ec84a89f27de4b357512"} Oct 30 16:53:48.315757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.315730 4828 generic.go:356] "Generic (PLEG): container finished" podID="5116f303-bd76-4921-bc75-1b8573200f6c" containerID="62346c4c94f8212ec0e91174743feb4f50c8e911265d59cfed54dde2fd3b2fc4" exitCode=0 Oct 30 16:53:48.315862 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.315763 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" event={"ID":"5116f303-bd76-4921-bc75-1b8573200f6c","Type":"ContainerDied","Data":"62346c4c94f8212ec0e91174743feb4f50c8e911265d59cfed54dde2fd3b2fc4"} Oct 30 16:53:48.469003 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.468963 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" Oct 30 16:53:48.471795 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.471773 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" Oct 30 16:53:48.579165 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.579086 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-util\" (UniqueName: \"kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-util\") pod \"0f16a076-c310-4aac-a65e-58ed25f29195\" (UID: \"0f16a076-c310-4aac-a65e-58ed25f29195\") " Oct 30 16:53:48.579165 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.579124 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-bundle\") pod \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\" (UID: \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\") " Oct 30 16:53:48.579354 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.579207 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-bundle\") pod \"0f16a076-c310-4aac-a65e-58ed25f29195\" (UID: \"0f16a076-c310-4aac-a65e-58ed25f29195\") " Oct 30 16:53:48.579354 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.579238 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-kube-api-access-rxd7h\" (UniqueName: \"kubernetes.io/projected/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-kube-api-access-rxd7h\") pod \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\" (UID: \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\") " Oct 30 16:53:48.579354 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.579268 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-util\" (UniqueName: \"kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-util\") pod \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\" (UID: \"2aad9f3e-ba60-436a-a1ae-5b848d339d9e\") " Oct 30 16:53:48.579354 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.579313 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f16a076-c310-4aac-a65e-58ed25f29195-kube-api-access-prbps\" (UniqueName: \"kubernetes.io/projected/0f16a076-c310-4aac-a65e-58ed25f29195-kube-api-access-prbps\") pod \"0f16a076-c310-4aac-a65e-58ed25f29195\" (UID: \"0f16a076-c310-4aac-a65e-58ed25f29195\") " Oct 30 16:53:48.579807 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.579775 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-bundle" pod "2aad9f3e-ba60-436a-a1ae-5b848d339d9e" (UID: "2aad9f3e-ba60-436a-a1ae-5b848d339d9e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:48.580063 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.580029 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-bundle" pod "0f16a076-c310-4aac-a65e-58ed25f29195" (UID: "0f16a076-c310-4aac-a65e-58ed25f29195"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:48.581534 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.581507 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-kube-api-access-rxd7h" pod "2aad9f3e-ba60-436a-a1ae-5b848d339d9e" (UID: "2aad9f3e-ba60-436a-a1ae-5b848d339d9e"). InnerVolumeSpecName "kube-api-access-rxd7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:48.581642 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.581623 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f16a076-c310-4aac-a65e-58ed25f29195-kube-api-access-prbps" pod "0f16a076-c310-4aac-a65e-58ed25f29195" (UID: "0f16a076-c310-4aac-a65e-58ed25f29195"). InnerVolumeSpecName "kube-api-access-prbps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:48.585920 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.585883 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-util" pod "0f16a076-c310-4aac-a65e-58ed25f29195" (UID: "0f16a076-c310-4aac-a65e-58ed25f29195"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:48.587573 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.587540 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-util" pod "2aad9f3e-ba60-436a-a1ae-5b848d339d9e" (UID: "2aad9f3e-ba60-436a-a1ae-5b848d339d9e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:48.680235 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.680180 4828 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-util\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:53:48.680235 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.680231 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prbps\" (UniqueName: \"kubernetes.io/projected/0f16a076-c310-4aac-a65e-58ed25f29195-kube-api-access-prbps\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:53:48.680235 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.680243 4828 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-util\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:53:48.680235 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.680252 4828 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-bundle\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:53:48.680501 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.680260 4828 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f16a076-c310-4aac-a65e-58ed25f29195-bundle\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:53:48.680501 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:48.680268 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxd7h\" (UniqueName: \"kubernetes.io/projected/2aad9f3e-ba60-436a-a1ae-5b848d339d9e-kube-api-access-rxd7h\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:53:49.322079 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.322049 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" Oct 30 16:53:49.322658 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.322083 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mlfgr" event={"ID":"2aad9f3e-ba60-436a-a1ae-5b848d339d9e","Type":"ContainerDied","Data":"b02c3538c1d15f384b314fb1edfaf2a88750a544ba3aec118b7d0d4aed9bc4cd"} Oct 30 16:53:49.322658 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.322129 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02c3538c1d15f384b314fb1edfaf2a88750a544ba3aec118b7d0d4aed9bc4cd" Oct 30 16:53:49.323790 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.323767 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" event={"ID":"0f16a076-c310-4aac-a65e-58ed25f29195","Type":"ContainerDied","Data":"aa30b21b3ae897189b2e795bd3287a4b1c58276988e2b264e0ad934b40a194b9"} Oct 30 16:53:49.323921 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.323803 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa30b21b3ae897189b2e795bd3287a4b1c58276988e2b264e0ad934b40a194b9" Oct 30 16:53:49.324601 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.324092 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847anrz7n" Oct 30 16:53:49.479236 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.479210 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" Oct 30 16:53:49.481368 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.481349 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" Oct 30 16:53:49.589033 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.588935 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-util\" (UniqueName: \"kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-util\") pod \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\" (UID: \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\") " Oct 30 16:53:49.589319 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.589295 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5116f303-bd76-4921-bc75-1b8573200f6c-kube-api-access-n8p7m\" (UniqueName: \"kubernetes.io/projected/5116f303-bd76-4921-bc75-1b8573200f6c-kube-api-access-n8p7m\") pod \"5116f303-bd76-4921-bc75-1b8573200f6c\" (UID: \"5116f303-bd76-4921-bc75-1b8573200f6c\") " Oct 30 16:53:49.589430 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.589412 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-bundle\") pod \"5116f303-bd76-4921-bc75-1b8573200f6c\" (UID: \"5116f303-bd76-4921-bc75-1b8573200f6c\") " Oct 30 16:53:49.589493 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.589463 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-bundle\" (UniqueName: \"kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-bundle\") pod \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\" (UID: \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\") " Oct 30 16:53:49.589548 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.589494 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-kube-api-access-kflmp\" (UniqueName: \"kubernetes.io/projected/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-kube-api-access-kflmp\") pod \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\" (UID: \"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37\") " Oct 30 16:53:49.589548 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.589534 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-util\" (UniqueName: \"kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-util\") pod \"5116f303-bd76-4921-bc75-1b8573200f6c\" (UID: \"5116f303-bd76-4921-bc75-1b8573200f6c\") " Oct 30 16:53:49.590452 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.590423 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-bundle" pod "5116f303-bd76-4921-bc75-1b8573200f6c" (UID: "5116f303-bd76-4921-bc75-1b8573200f6c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:49.590757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.590720 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-bundle" pod "12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37" (UID: "12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:49.592694 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.592670 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5116f303-bd76-4921-bc75-1b8573200f6c-kube-api-access-n8p7m" pod "5116f303-bd76-4921-bc75-1b8573200f6c" (UID: "5116f303-bd76-4921-bc75-1b8573200f6c"). InnerVolumeSpecName "kube-api-access-n8p7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:49.592979 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.592951 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-kube-api-access-kflmp" pod "12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37" (UID: "12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37"). InnerVolumeSpecName "kube-api-access-kflmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:49.596742 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.596710 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-util" pod "5116f303-bd76-4921-bc75-1b8573200f6c" (UID: "5116f303-bd76-4921-bc75-1b8573200f6c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:49.596874 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.596851 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-util" pod "12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37" (UID: "12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:49.690722 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.690679 4828 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-util\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:53:49.690722 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.690715 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8p7m\" (UniqueName: \"kubernetes.io/projected/5116f303-bd76-4921-bc75-1b8573200f6c-kube-api-access-n8p7m\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:53:49.690722 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.690725 4828 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-bundle\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:53:49.690722 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.690733 4828 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-bundle\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:53:49.691017 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.690745 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kflmp\" (UniqueName: \"kubernetes.io/projected/12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37-kube-api-access-kflmp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:53:49.691017 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:49.690753 4828 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5116f303-bd76-4921-bc75-1b8573200f6c-util\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:53:50.330291 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:50.330260 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" Oct 30 16:53:50.330741 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:50.330286 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjlcb" event={"ID":"5116f303-bd76-4921-bc75-1b8573200f6c","Type":"ContainerDied","Data":"bd7ca4b325220e7e371bf9f36c5cea58edb8d59b1f8a81c38dfe86dd955c7f11"} Oct 30 16:53:50.330741 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:50.330368 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd7ca4b325220e7e371bf9f36c5cea58edb8d59b1f8a81c38dfe86dd955c7f11" Oct 30 16:53:50.332309 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:50.332286 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" event={"ID":"12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37","Type":"ContainerDied","Data":"22aadb6525c27fd9efd0b5a3a004843bc1e4233a6ef118963c83214a35680663"} Oct 30 16:53:50.332430 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:50.332315 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw2t8s" Oct 30 16:53:50.332430 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:50.332320 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22aadb6525c27fd9efd0b5a3a004843bc1e4233a6ef118963c83214a35680663" Oct 30 16:53:52.485072 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:52.485031 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:53.103692 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:53.103652 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:53.272379 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:53.272334 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:54.086170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:54.086114 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:57.015764 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.015724 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b"] Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016079 4828 cpu_manager.go:414] "containerMap: removing container" podUID="5116f303-bd76-4921-bc75-1b8573200f6c" containerName="extract" Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016091 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5116f303-bd76-4921-bc75-1b8573200f6c" containerName="extract" Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016106 4828 cpu_manager.go:414] "containerMap: removing container" podUID="2aad9f3e-ba60-436a-a1ae-5b848d339d9e" containerName="pull" Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016111 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aad9f3e-ba60-436a-a1ae-5b848d339d9e" containerName="pull" Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016120 4828 cpu_manager.go:414] "containerMap: removing container" podUID="0f16a076-c310-4aac-a65e-58ed25f29195" containerName="pull" Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016126 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f16a076-c310-4aac-a65e-58ed25f29195" containerName="pull" Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016134 4828 cpu_manager.go:414] "containerMap: removing container" podUID="0f16a076-c310-4aac-a65e-58ed25f29195" containerName="util" Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016139 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f16a076-c310-4aac-a65e-58ed25f29195" containerName="util" Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016146 4828 cpu_manager.go:414] "containerMap: removing container" podUID="2aad9f3e-ba60-436a-a1ae-5b848d339d9e" containerName="util" Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016150 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aad9f3e-ba60-436a-a1ae-5b848d339d9e" containerName="util" Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016155 4828 cpu_manager.go:414] "containerMap: removing container" podUID="12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37" containerName="util" Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016161 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37" containerName="util" Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016171 4828 cpu_manager.go:414] "containerMap: removing container" podUID="12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37" containerName="extract" Oct 30 16:53:57.016170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016177 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37" containerName="extract" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016184 4828 cpu_manager.go:414] "containerMap: removing container" podUID="12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37" containerName="pull" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016189 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37" containerName="pull" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016196 4828 cpu_manager.go:414] "containerMap: removing container" podUID="2aad9f3e-ba60-436a-a1ae-5b848d339d9e" containerName="extract" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016200 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aad9f3e-ba60-436a-a1ae-5b848d339d9e" containerName="extract" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016211 4828 cpu_manager.go:414] "containerMap: removing container" podUID="5116f303-bd76-4921-bc75-1b8573200f6c" containerName="util" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016215 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5116f303-bd76-4921-bc75-1b8573200f6c" containerName="util" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016222 4828 cpu_manager.go:414] "containerMap: removing container" podUID="5116f303-bd76-4921-bc75-1b8573200f6c" containerName="pull" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016226 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5116f303-bd76-4921-bc75-1b8573200f6c" containerName="pull" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016232 4828 cpu_manager.go:414] "containerMap: removing container" podUID="0f16a076-c310-4aac-a65e-58ed25f29195" containerName="extract" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016236 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f16a076-c310-4aac-a65e-58ed25f29195" containerName="extract" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016290 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b19b1f-2cd9-4cee-a0d4-cdc54ad04b37" containerName="extract" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016298 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aad9f3e-ba60-436a-a1ae-5b848d339d9e" containerName="extract" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016306 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f16a076-c310-4aac-a65e-58ed25f29195" containerName="extract" Oct 30 16:53:57.016570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.016313 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="5116f303-bd76-4921-bc75-1b8573200f6c" containerName="extract" Oct 30 16:53:57.020506 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.020489 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:53:57.034531 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.034500 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b"] Oct 30 16:53:57.036970 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.036942 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 16:53:57.037081 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.036953 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 16:53:57.045949 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.045928 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jdb28\"" Oct 30 16:53:57.154343 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.154309 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/673ec9d5-350f-4a29-bd9e-1120d88a9768-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-mqm8b\" (UID: \"673ec9d5-350f-4a29-bd9e-1120d88a9768\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:53:57.154517 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.154375 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhlzj\" (UniqueName: \"kubernetes.io/projected/673ec9d5-350f-4a29-bd9e-1120d88a9768-kube-api-access-bhlzj\") pod \"kuadrant-operator-controller-manager-6b6fb56846-mqm8b\" (UID: \"673ec9d5-350f-4a29-bd9e-1120d88a9768\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:53:57.167744 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.167710 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:57.255307 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.255266 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlzj\" (UniqueName: \"kubernetes.io/projected/673ec9d5-350f-4a29-bd9e-1120d88a9768-kube-api-access-bhlzj\") pod \"kuadrant-operator-controller-manager-6b6fb56846-mqm8b\" (UID: \"673ec9d5-350f-4a29-bd9e-1120d88a9768\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:53:57.255497 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.255336 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/673ec9d5-350f-4a29-bd9e-1120d88a9768-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-mqm8b\" (UID: \"673ec9d5-350f-4a29-bd9e-1120d88a9768\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:53:57.255672 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.255655 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/673ec9d5-350f-4a29-bd9e-1120d88a9768-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-mqm8b\" (UID: \"673ec9d5-350f-4a29-bd9e-1120d88a9768\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:53:57.286317 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.286242 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhlzj\" (UniqueName: \"kubernetes.io/projected/673ec9d5-350f-4a29-bd9e-1120d88a9768-kube-api-access-bhlzj\") pod \"kuadrant-operator-controller-manager-6b6fb56846-mqm8b\" (UID: \"673ec9d5-350f-4a29-bd9e-1120d88a9768\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:53:57.333168 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.333132 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:53:57.339649 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.339617 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:57.673935 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.665287 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b"] Oct 30 16:53:57.673935 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:53:57.668755 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673ec9d5_350f_4a29_bd9e_1120d88a9768.slice/crio-a877b1d1f0e34a4571be490eecd34d0df52e6b48615a62216ec202c2e5c028f5 WatchSource:0}: Error finding container a877b1d1f0e34a4571be490eecd34d0df52e6b48615a62216ec202c2e5c028f5: Status 404 returned error can't find the container with id a877b1d1f0e34a4571be490eecd34d0df52e6b48615a62216ec202c2e5c028f5 Oct 30 16:53:57.719562 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:57.719515 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:58.372618 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:58.372581 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" event={"ID":"673ec9d5-350f-4a29-bd9e-1120d88a9768","Type":"ContainerStarted","Data":"a877b1d1f0e34a4571be490eecd34d0df52e6b48615a62216ec202c2e5c028f5"} Oct 30 16:53:58.902150 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:58.902116 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-2xqwn"] Oct 30 16:53:58.906024 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:58.905997 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-2xqwn" Oct 30 16:53:58.925629 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:58.925600 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5xj8k\"" Oct 30 16:53:58.939573 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:58.939542 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 16:53:58.942662 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:58.942632 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-2xqwn"] Oct 30 16:53:58.972774 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:58.972742 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nptd\" (UniqueName: \"kubernetes.io/projected/215337ca-676e-4628-ab89-9da3fd69ae17-kube-api-access-4nptd\") pod \"dns-operator-controller-manager-65bfd79598-2xqwn\" (UID: \"215337ca-676e-4628-ab89-9da3fd69ae17\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-2xqwn" Oct 30 16:53:59.073935 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:59.073552 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nptd\" (UniqueName: \"kubernetes.io/projected/215337ca-676e-4628-ab89-9da3fd69ae17-kube-api-access-4nptd\") pod \"dns-operator-controller-manager-65bfd79598-2xqwn\" (UID: \"215337ca-676e-4628-ab89-9da3fd69ae17\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-2xqwn" Oct 30 16:53:59.105886 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:59.105844 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nptd\" (UniqueName: \"kubernetes.io/projected/215337ca-676e-4628-ab89-9da3fd69ae17-kube-api-access-4nptd\") pod \"dns-operator-controller-manager-65bfd79598-2xqwn\" (UID: \"215337ca-676e-4628-ab89-9da3fd69ae17\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-2xqwn" Oct 30 16:53:59.223576 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:59.223405 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-2xqwn" Oct 30 16:53:59.267963 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:59.267820 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:59.533133 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:59.533078 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:53:59.545018 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:59.544991 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-2xqwn"] Oct 30 16:53:59.545700 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:53:59.545670 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215337ca_676e_4628_ab89_9da3fd69ae17.slice/crio-532e2a28c4544c172676a64e42a7d5d76582af2509532720213e7bd41dec197c WatchSource:0}: Error finding container 532e2a28c4544c172676a64e42a7d5d76582af2509532720213e7bd41dec197c: Status 404 returned error can't find the container with id 532e2a28c4544c172676a64e42a7d5d76582af2509532720213e7bd41dec197c Oct 30 16:53:59.700536 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:53:59.700500 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:00.386134 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:00.386096 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-2xqwn" event={"ID":"215337ca-676e-4628-ab89-9da3fd69ae17","Type":"ContainerStarted","Data":"532e2a28c4544c172676a64e42a7d5d76582af2509532720213e7bd41dec197c"} Oct 30 16:54:01.404460 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:01.404405 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-2xqwn" event={"ID":"215337ca-676e-4628-ab89-9da3fd69ae17","Type":"ContainerStarted","Data":"4530e82680aa3a668f2231c6d5af527df70d93651c4a6b26de78b714e04a382f"} Oct 30 16:54:01.404971 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:01.404641 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-2xqwn" Oct 30 16:54:01.453071 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:01.453009 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-2xqwn" podStartSLOduration=2.147572257 podStartE2EDuration="3.452988563s" totalImagesPullingTime="1.305416306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:01.450113231 +0000 UTC m=+419.095623429" watchObservedRunningTime="2026-10-30 16:54:01.452988563 +0000 UTC m=+419.098498745" Oct 30 16:54:02.334043 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:02.334005 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:02.411176 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:02.411126 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" event={"ID":"673ec9d5-350f-4a29-bd9e-1120d88a9768","Type":"ContainerStarted","Data":"a6e2800b185ef1a6ed4a4b91156ec3da4c7039c782691b3e331a0aa5f45ecd83"} Oct 30 16:54:02.411655 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:02.411208 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:54:02.493397 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:02.493345 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" podStartSLOduration=2.579676223 podStartE2EDuration="6.49332982s" totalImagesPullingTime="3.913653597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:02.47792239 +0000 UTC m=+420.123432571" watchObservedRunningTime="2026-10-30 16:54:02.49332982 +0000 UTC m=+420.138840000" Oct 30 16:54:02.550346 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:02.550311 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:03.907322 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:03.907287 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc"] Oct 30 16:54:03.911143 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:03.911122 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" Oct 30 16:54:03.930041 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:03.930017 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-f788c\"" Oct 30 16:54:03.932095 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:03.932072 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc"] Oct 30 16:54:04.032283 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:04.032249 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb7bq\" (UniqueName: \"kubernetes.io/projected/a084bdb7-f7be-4174-ac84-b3aba11e93a6-kube-api-access-tb7bq\") pod \"limitador-operator-controller-manager-856dd584c5-fp9rc\" (UID: \"a084bdb7-f7be-4174-ac84-b3aba11e93a6\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" Oct 30 16:54:04.132681 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:04.132639 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb7bq\" (UniqueName: \"kubernetes.io/projected/a084bdb7-f7be-4174-ac84-b3aba11e93a6-kube-api-access-tb7bq\") pod \"limitador-operator-controller-manager-856dd584c5-fp9rc\" (UID: \"a084bdb7-f7be-4174-ac84-b3aba11e93a6\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" Oct 30 16:54:04.158225 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:04.158148 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb7bq\" (UniqueName: \"kubernetes.io/projected/a084bdb7-f7be-4174-ac84-b3aba11e93a6-kube-api-access-tb7bq\") pod \"limitador-operator-controller-manager-856dd584c5-fp9rc\" (UID: \"a084bdb7-f7be-4174-ac84-b3aba11e93a6\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" Oct 30 16:54:04.223738 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:04.223703 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" Oct 30 16:54:04.301212 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:04.301161 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:04.529570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:04.526040 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc"] Oct 30 16:54:04.913104 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:04.913012 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:05.067410 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:05.067370 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:05.426450 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:05.426410 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" event={"ID":"a084bdb7-f7be-4174-ac84-b3aba11e93a6","Type":"ContainerStarted","Data":"c617806f85186830d906a07ee9b0d45eaa417e93c87d95fa1f50963c6580341e"} Oct 30 16:54:07.888872 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:07.888840 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:07.910337 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:54:07.910304 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:54:08.034819 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:08.034782 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:08.279942 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:08.279892 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:08.442176 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:08.442131 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:08.673425 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:08.673331 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:08.887841 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:08.887803 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:12.413702 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:12.413661 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-2xqwn" Oct 30 16:54:13.418477 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:13.418445 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:54:14.391124 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:14.391089 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:14.555454 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:14.555420 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:14.888985 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:14.888944 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg"] Oct 30 16:54:14.893782 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:14.893757 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" Oct 30 16:54:14.912096 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:14.912064 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg"] Oct 30 16:54:15.024070 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:15.024031 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d91990d-34b7-47eb-bcd0-ba3973a58615-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-4pxbg\" (UID: \"5d91990d-34b7-47eb-bcd0-ba3973a58615\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" Oct 30 16:54:15.024241 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:15.024078 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w9v5\" (UniqueName: \"kubernetes.io/projected/5d91990d-34b7-47eb-bcd0-ba3973a58615-kube-api-access-2w9v5\") pod \"kuadrant-operator-controller-manager-6b76784d48-4pxbg\" (UID: \"5d91990d-34b7-47eb-bcd0-ba3973a58615\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" Oct 30 16:54:15.125105 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:15.125067 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d91990d-34b7-47eb-bcd0-ba3973a58615-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-4pxbg\" (UID: \"5d91990d-34b7-47eb-bcd0-ba3973a58615\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" Oct 30 16:54:15.125273 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:15.125116 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w9v5\" (UniqueName: \"kubernetes.io/projected/5d91990d-34b7-47eb-bcd0-ba3973a58615-kube-api-access-2w9v5\") pod \"kuadrant-operator-controller-manager-6b76784d48-4pxbg\" (UID: \"5d91990d-34b7-47eb-bcd0-ba3973a58615\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" Oct 30 16:54:15.125544 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:15.125521 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d91990d-34b7-47eb-bcd0-ba3973a58615-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-4pxbg\" (UID: \"5d91990d-34b7-47eb-bcd0-ba3973a58615\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" Oct 30 16:54:15.151623 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:15.151545 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w9v5\" (UniqueName: \"kubernetes.io/projected/5d91990d-34b7-47eb-bcd0-ba3973a58615-kube-api-access-2w9v5\") pod \"kuadrant-operator-controller-manager-6b76784d48-4pxbg\" (UID: \"5d91990d-34b7-47eb-bcd0-ba3973a58615\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" Oct 30 16:54:15.172131 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:15.172098 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:15.207366 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:15.207332 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" Oct 30 16:54:15.335009 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:15.334970 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:15.538660 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:15.538626 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg"] Oct 30 16:54:15.552388 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:54:15.552355 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d91990d_34b7_47eb_bcd0_ba3973a58615.slice/crio-6f5c9f3cec641f4e22954920c31ae8ee92e9e4f6c53e82e5ec710de5b99526e4 WatchSource:0}: Error finding container 6f5c9f3cec641f4e22954920c31ae8ee92e9e4f6c53e82e5ec710de5b99526e4: Status 404 returned error can't find the container with id 6f5c9f3cec641f4e22954920c31ae8ee92e9e4f6c53e82e5ec710de5b99526e4 Oct 30 16:54:16.488963 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:16.488927 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" event={"ID":"5d91990d-34b7-47eb-bcd0-ba3973a58615","Type":"ContainerStarted","Data":"4ce9ebd7f531e0576be41f3409c3c310dfb6899e8c4e20c02b0c2cde211e6205"} Oct 30 16:54:16.489384 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:16.488987 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" Oct 30 16:54:16.489384 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:16.489013 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" event={"ID":"5d91990d-34b7-47eb-bcd0-ba3973a58615","Type":"ContainerStarted","Data":"6f5c9f3cec641f4e22954920c31ae8ee92e9e4f6c53e82e5ec710de5b99526e4"} Oct 30 16:54:16.489384 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:16.489188 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" event={"ID":"a084bdb7-f7be-4174-ac84-b3aba11e93a6","Type":"ContainerStarted","Data":"67a710e46a88024f925fe02a020dda7b9274a1f65cce6114abce11356fb12ebf"} Oct 30 16:54:16.489384 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:16.489318 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" Oct 30 16:54:16.569917 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:16.569847 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" podStartSLOduration=2.569831738 podStartE2EDuration="2.569831738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:16.567944143 +0000 UTC m=+434.213454324" watchObservedRunningTime="2026-10-30 16:54:16.569831738 +0000 UTC m=+434.215341930" Oct 30 16:54:16.617839 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:16.617764 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" podStartSLOduration=1.7661736019999998 podStartE2EDuration="13.617742919s" totalImagesPullingTime="11.851569317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:16.615964485 +0000 UTC m=+434.261474665" watchObservedRunningTime="2026-10-30 16:54:16.617742919 +0000 UTC m=+434.263253102" Oct 30 16:54:17.268729 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.268689 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b"] Oct 30 16:54:17.269106 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.269042 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" podUID="673ec9d5-350f-4a29-bd9e-1120d88a9768" containerName="manager" containerID="cri-o://a6e2800b185ef1a6ed4a4b91156ec3da4c7039c782691b3e331a0aa5f45ecd83" gracePeriod=2 Oct 30 16:54:17.271786 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.271757 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b"] Oct 30 16:54:17.284213 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:54:17.284169 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-mqm8b\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="673ec9d5-350f-4a29-bd9e-1120d88a9768" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:54:17.333494 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.333461 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j"] Oct 30 16:54:17.333866 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.333850 4828 cpu_manager.go:414] "containerMap: removing container" podUID="673ec9d5-350f-4a29-bd9e-1120d88a9768" containerName="manager" Oct 30 16:54:17.333947 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.333867 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="673ec9d5-350f-4a29-bd9e-1120d88a9768" containerName="manager" Oct 30 16:54:17.333992 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.333968 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="673ec9d5-350f-4a29-bd9e-1120d88a9768" containerName="manager" Oct 30 16:54:17.338503 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.338471 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" Oct 30 16:54:17.340300 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.340271 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg"] Oct 30 16:54:17.344574 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.344508 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzk6f\" (UniqueName: \"kubernetes.io/projected/c3c4f0b8-5630-43bf-8a73-01a21547f55c-kube-api-access-mzk6f\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wm96j\" (UID: \"c3c4f0b8-5630-43bf-8a73-01a21547f55c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" Oct 30 16:54:17.344746 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.344616 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3c4f0b8-5630-43bf-8a73-01a21547f55c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wm96j\" (UID: \"c3c4f0b8-5630-43bf-8a73-01a21547f55c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" Oct 30 16:54:17.348938 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.348365 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg"] Oct 30 16:54:17.373755 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.373722 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc"] Oct 30 16:54:17.398611 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.398576 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j"] Oct 30 16:54:17.411354 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.411322 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc"] Oct 30 16:54:17.415242 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.414856 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n"] Oct 30 16:54:17.415389 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.415254 4828 cpu_manager.go:414] "containerMap: removing container" podUID="5d91990d-34b7-47eb-bcd0-ba3973a58615" containerName="manager" Oct 30 16:54:17.415389 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.415272 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d91990d-34b7-47eb-bcd0-ba3973a58615" containerName="manager" Oct 30 16:54:17.415389 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.415292 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a084bdb7-f7be-4174-ac84-b3aba11e93a6" containerName="manager" Oct 30 16:54:17.415389 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.415299 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a084bdb7-f7be-4174-ac84-b3aba11e93a6" containerName="manager" Oct 30 16:54:17.415542 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.415391 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d91990d-34b7-47eb-bcd0-ba3973a58615" containerName="manager" Oct 30 16:54:17.415542 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.415400 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a084bdb7-f7be-4174-ac84-b3aba11e93a6" containerName="manager" Oct 30 16:54:17.418328 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.418308 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" Oct 30 16:54:17.427170 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:54:17.427132 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-mqm8b\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="673ec9d5-350f-4a29-bd9e-1120d88a9768" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:54:17.442678 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:54:17.442645 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-mqm8b\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="673ec9d5-350f-4a29-bd9e-1120d88a9768" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:54:17.445150 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.445113 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzk6f\" (UniqueName: \"kubernetes.io/projected/c3c4f0b8-5630-43bf-8a73-01a21547f55c-kube-api-access-mzk6f\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wm96j\" (UID: \"c3c4f0b8-5630-43bf-8a73-01a21547f55c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" Oct 30 16:54:17.445297 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.445189 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3c4f0b8-5630-43bf-8a73-01a21547f55c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wm96j\" (UID: \"c3c4f0b8-5630-43bf-8a73-01a21547f55c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" Oct 30 16:54:17.445627 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.445580 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3c4f0b8-5630-43bf-8a73-01a21547f55c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wm96j\" (UID: \"c3c4f0b8-5630-43bf-8a73-01a21547f55c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" Oct 30 16:54:17.447805 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.447780 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n"] Oct 30 16:54:17.485884 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.485853 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzk6f\" (UniqueName: \"kubernetes.io/projected/c3c4f0b8-5630-43bf-8a73-01a21547f55c-kube-api-access-mzk6f\") pod \"kuadrant-operator-controller-manager-6b6fb56846-wm96j\" (UID: \"c3c4f0b8-5630-43bf-8a73-01a21547f55c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" Oct 30 16:54:17.496365 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.496336 4828 generic.go:356] "Generic (PLEG): container finished" podID="673ec9d5-350f-4a29-bd9e-1120d88a9768" containerID="a6e2800b185ef1a6ed4a4b91156ec3da4c7039c782691b3e331a0aa5f45ecd83" exitCode=0 Oct 30 16:54:17.546271 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.546240 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x9vc\" (UniqueName: \"kubernetes.io/projected/5238d15a-6a60-4e03-bb72-9daec5a28e9b-kube-api-access-8x9vc\") pod \"kuadrant-operator-controller-manager-6b76784d48-vtv9n\" (UID: \"5238d15a-6a60-4e03-bb72-9daec5a28e9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" Oct 30 16:54:17.546416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.546358 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5238d15a-6a60-4e03-bb72-9daec5a28e9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-vtv9n\" (UID: \"5238d15a-6a60-4e03-bb72-9daec5a28e9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" Oct 30 16:54:17.553450 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.553427 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:54:17.568540 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:54:17.568492 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-mqm8b\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="673ec9d5-350f-4a29-bd9e-1120d88a9768" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:54:17.646745 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.646711 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673ec9d5-350f-4a29-bd9e-1120d88a9768-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/673ec9d5-350f-4a29-bd9e-1120d88a9768-extensions-socket-volume\") pod \"673ec9d5-350f-4a29-bd9e-1120d88a9768\" (UID: \"673ec9d5-350f-4a29-bd9e-1120d88a9768\") " Oct 30 16:54:17.646887 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.646789 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673ec9d5-350f-4a29-bd9e-1120d88a9768-kube-api-access-bhlzj\" (UniqueName: \"kubernetes.io/projected/673ec9d5-350f-4a29-bd9e-1120d88a9768-kube-api-access-bhlzj\") pod \"673ec9d5-350f-4a29-bd9e-1120d88a9768\" (UID: \"673ec9d5-350f-4a29-bd9e-1120d88a9768\") " Oct 30 16:54:17.646887 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.646869 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5238d15a-6a60-4e03-bb72-9daec5a28e9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-vtv9n\" (UID: \"5238d15a-6a60-4e03-bb72-9daec5a28e9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" Oct 30 16:54:17.647019 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.646973 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x9vc\" (UniqueName: \"kubernetes.io/projected/5238d15a-6a60-4e03-bb72-9daec5a28e9b-kube-api-access-8x9vc\") pod \"kuadrant-operator-controller-manager-6b76784d48-vtv9n\" (UID: \"5238d15a-6a60-4e03-bb72-9daec5a28e9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" Oct 30 16:54:17.647133 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.647110 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673ec9d5-350f-4a29-bd9e-1120d88a9768-extensions-socket-volume" pod "673ec9d5-350f-4a29-bd9e-1120d88a9768" (UID: "673ec9d5-350f-4a29-bd9e-1120d88a9768"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:54:17.647306 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.647283 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5238d15a-6a60-4e03-bb72-9daec5a28e9b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-vtv9n\" (UID: \"5238d15a-6a60-4e03-bb72-9daec5a28e9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" Oct 30 16:54:17.648968 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.648946 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673ec9d5-350f-4a29-bd9e-1120d88a9768-kube-api-access-bhlzj" pod "673ec9d5-350f-4a29-bd9e-1120d88a9768" (UID: "673ec9d5-350f-4a29-bd9e-1120d88a9768"). InnerVolumeSpecName "kube-api-access-bhlzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:54:17.658638 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.658615 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" Oct 30 16:54:17.672181 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.672156 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x9vc\" (UniqueName: \"kubernetes.io/projected/5238d15a-6a60-4e03-bb72-9daec5a28e9b-kube-api-access-8x9vc\") pod \"kuadrant-operator-controller-manager-6b76784d48-vtv9n\" (UID: \"5238d15a-6a60-4e03-bb72-9daec5a28e9b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" Oct 30 16:54:17.733013 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.732981 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" Oct 30 16:54:17.747557 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.747408 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhlzj\" (UniqueName: \"kubernetes.io/projected/673ec9d5-350f-4a29-bd9e-1120d88a9768-kube-api-access-bhlzj\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:54:17.747557 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.747433 4828 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/673ec9d5-350f-4a29-bd9e-1120d88a9768-extensions-socket-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:54:17.970690 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:17.970666 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j"] Oct 30 16:54:17.972524 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:54:17.972491 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c4f0b8_5630_43bf_8a73_01a21547f55c.slice/crio-1f8b3ca98d15014637cbb53262914648d1feeac568048f6ae61f55a733865ad5 WatchSource:0}: Error finding container 1f8b3ca98d15014637cbb53262914648d1feeac568048f6ae61f55a733865ad5: Status 404 returned error can't find the container with id 1f8b3ca98d15014637cbb53262914648d1feeac568048f6ae61f55a733865ad5 Oct 30 16:54:18.040326 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.040298 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n"] Oct 30 16:54:18.041509 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:54:18.041485 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5238d15a_6a60_4e03_bb72_9daec5a28e9b.slice/crio-9a1cc0f16b828cfda5fc63ed4ebafd05f5ea3109cb403903bb3cda3d02d5139f WatchSource:0}: Error finding container 9a1cc0f16b828cfda5fc63ed4ebafd05f5ea3109cb403903bb3cda3d02d5139f: Status 404 returned error can't find the container with id 9a1cc0f16b828cfda5fc63ed4ebafd05f5ea3109cb403903bb3cda3d02d5139f Oct 30 16:54:18.488656 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.488573 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="673ec9d5-350f-4a29-bd9e-1120d88a9768" path="/var/lib/kubelet/pods/673ec9d5-350f-4a29-bd9e-1120d88a9768/volumes" Oct 30 16:54:18.503499 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.503456 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" event={"ID":"5238d15a-6a60-4e03-bb72-9daec5a28e9b","Type":"ContainerStarted","Data":"7632ccbb649923c7c330e3853a38ed5543f2f98836a43c04cc69aa0cf1744aa0"} Oct 30 16:54:18.503886 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.503529 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" Oct 30 16:54:18.503886 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.503555 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" event={"ID":"5238d15a-6a60-4e03-bb72-9daec5a28e9b","Type":"ContainerStarted","Data":"9a1cc0f16b828cfda5fc63ed4ebafd05f5ea3109cb403903bb3cda3d02d5139f"} Oct 30 16:54:18.504867 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.504850 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-mqm8b" Oct 30 16:54:18.504933 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.504861 4828 scope.go:122] "RemoveContainer" containerID="a6e2800b185ef1a6ed4a4b91156ec3da4c7039c782691b3e331a0aa5f45ecd83" Oct 30 16:54:18.507182 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.507160 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" event={"ID":"c3c4f0b8-5630-43bf-8a73-01a21547f55c","Type":"ContainerStarted","Data":"fd54a936e74b3a4a418f00601b788b86649ded874725508b77ea6fb46f458211"} Oct 30 16:54:18.507285 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.507198 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" event={"ID":"c3c4f0b8-5630-43bf-8a73-01a21547f55c","Type":"ContainerStarted","Data":"1f8b3ca98d15014637cbb53262914648d1feeac568048f6ae61f55a733865ad5"} Oct 30 16:54:18.507285 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.507251 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" podUID="5d91990d-34b7-47eb-bcd0-ba3973a58615" containerName="manager" containerID="cri-o://4ce9ebd7f531e0576be41f3409c3c310dfb6899e8c4e20c02b0c2cde211e6205" gracePeriod=2 Oct 30 16:54:18.507388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.507302 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" podUID="a084bdb7-f7be-4174-ac84-b3aba11e93a6" containerName="manager" containerID="cri-o://67a710e46a88024f925fe02a020dda7b9274a1f65cce6114abce11356fb12ebf" gracePeriod=2 Oct 30 16:54:18.554472 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.554405 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" podStartSLOduration=1.554390194 podStartE2EDuration="1.554390194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:18.552316969 +0000 UTC m=+436.197827149" watchObservedRunningTime="2026-10-30 16:54:18.554390194 +0000 UTC m=+436.199900374" Oct 30 16:54:18.606044 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.605989 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" podStartSLOduration=1.605975971 podStartE2EDuration="1.605975971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:18.605397896 +0000 UTC m=+436.250908076" watchObservedRunningTime="2026-10-30 16:54:18.605975971 +0000 UTC m=+436.251486150" Oct 30 16:54:18.782912 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.782866 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" Oct 30 16:54:18.785666 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.785636 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" Oct 30 16:54:18.856784 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.856745 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d91990d-34b7-47eb-bcd0-ba3973a58615-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d91990d-34b7-47eb-bcd0-ba3973a58615-extensions-socket-volume\") pod \"5d91990d-34b7-47eb-bcd0-ba3973a58615\" (UID: \"5d91990d-34b7-47eb-bcd0-ba3973a58615\") " Oct 30 16:54:18.856992 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.856796 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a084bdb7-f7be-4174-ac84-b3aba11e93a6-kube-api-access-tb7bq\" (UniqueName: \"kubernetes.io/projected/a084bdb7-f7be-4174-ac84-b3aba11e93a6-kube-api-access-tb7bq\") pod \"a084bdb7-f7be-4174-ac84-b3aba11e93a6\" (UID: \"a084bdb7-f7be-4174-ac84-b3aba11e93a6\") " Oct 30 16:54:18.856992 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.856816 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d91990d-34b7-47eb-bcd0-ba3973a58615-kube-api-access-2w9v5\" (UniqueName: \"kubernetes.io/projected/5d91990d-34b7-47eb-bcd0-ba3973a58615-kube-api-access-2w9v5\") pod \"5d91990d-34b7-47eb-bcd0-ba3973a58615\" (UID: \"5d91990d-34b7-47eb-bcd0-ba3973a58615\") " Oct 30 16:54:18.857190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.857165 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d91990d-34b7-47eb-bcd0-ba3973a58615-extensions-socket-volume" pod "5d91990d-34b7-47eb-bcd0-ba3973a58615" (UID: "5d91990d-34b7-47eb-bcd0-ba3973a58615"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:54:18.858956 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.858919 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a084bdb7-f7be-4174-ac84-b3aba11e93a6-kube-api-access-tb7bq" pod "a084bdb7-f7be-4174-ac84-b3aba11e93a6" (UID: "a084bdb7-f7be-4174-ac84-b3aba11e93a6"). InnerVolumeSpecName "kube-api-access-tb7bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:54:18.858956 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.858935 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d91990d-34b7-47eb-bcd0-ba3973a58615-kube-api-access-2w9v5" pod "5d91990d-34b7-47eb-bcd0-ba3973a58615" (UID: "5d91990d-34b7-47eb-bcd0-ba3973a58615"). InnerVolumeSpecName "kube-api-access-2w9v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:54:18.957860 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.957822 4828 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d91990d-34b7-47eb-bcd0-ba3973a58615-extensions-socket-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:54:18.957860 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.957853 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb7bq\" (UniqueName: \"kubernetes.io/projected/a084bdb7-f7be-4174-ac84-b3aba11e93a6-kube-api-access-tb7bq\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:54:18.957860 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:18.957863 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w9v5\" (UniqueName: \"kubernetes.io/projected/5d91990d-34b7-47eb-bcd0-ba3973a58615-kube-api-access-2w9v5\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:54:19.513703 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:19.513666 4828 generic.go:356] "Generic (PLEG): container finished" podID="a084bdb7-f7be-4174-ac84-b3aba11e93a6" containerID="67a710e46a88024f925fe02a020dda7b9274a1f65cce6114abce11356fb12ebf" exitCode=0 Oct 30 16:54:19.514152 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:19.513729 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" Oct 30 16:54:19.514152 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:19.513782 4828 scope.go:122] "RemoveContainer" containerID="67a710e46a88024f925fe02a020dda7b9274a1f65cce6114abce11356fb12ebf" Oct 30 16:54:19.516778 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:19.516745 4828 generic.go:356] "Generic (PLEG): container finished" podID="5d91990d-34b7-47eb-bcd0-ba3973a58615" containerID="4ce9ebd7f531e0576be41f3409c3c310dfb6899e8c4e20c02b0c2cde211e6205" exitCode=0 Oct 30 16:54:19.516933 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:19.516792 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" Oct 30 16:54:19.517284 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:19.517261 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" Oct 30 16:54:19.523646 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:19.523629 4828 scope.go:122] "RemoveContainer" containerID="67a710e46a88024f925fe02a020dda7b9274a1f65cce6114abce11356fb12ebf" Oct 30 16:54:19.523941 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:54:19.523921 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a710e46a88024f925fe02a020dda7b9274a1f65cce6114abce11356fb12ebf\": container with ID starting with 67a710e46a88024f925fe02a020dda7b9274a1f65cce6114abce11356fb12ebf not found: ID does not exist" containerID="67a710e46a88024f925fe02a020dda7b9274a1f65cce6114abce11356fb12ebf" Oct 30 16:54:19.524041 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:19.523950 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a710e46a88024f925fe02a020dda7b9274a1f65cce6114abce11356fb12ebf"} err="failed to get container status \"67a710e46a88024f925fe02a020dda7b9274a1f65cce6114abce11356fb12ebf\": rpc error: code = NotFound desc = could not find container \"67a710e46a88024f925fe02a020dda7b9274a1f65cce6114abce11356fb12ebf\": container with ID starting with 67a710e46a88024f925fe02a020dda7b9274a1f65cce6114abce11356fb12ebf not found: ID does not exist" Oct 30 16:54:19.524041 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:19.523972 4828 scope.go:122] "RemoveContainer" containerID="4ce9ebd7f531e0576be41f3409c3c310dfb6899e8c4e20c02b0c2cde211e6205" Oct 30 16:54:19.528811 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:54:19.528778 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-fp9rc\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="a084bdb7-f7be-4174-ac84-b3aba11e93a6" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" Oct 30 16:54:19.532274 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:19.532256 4828 scope.go:122] "RemoveContainer" containerID="4ce9ebd7f531e0576be41f3409c3c310dfb6899e8c4e20c02b0c2cde211e6205" Oct 30 16:54:19.532540 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:54:19.532520 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce9ebd7f531e0576be41f3409c3c310dfb6899e8c4e20c02b0c2cde211e6205\": container with ID starting with 4ce9ebd7f531e0576be41f3409c3c310dfb6899e8c4e20c02b0c2cde211e6205 not found: ID does not exist" containerID="4ce9ebd7f531e0576be41f3409c3c310dfb6899e8c4e20c02b0c2cde211e6205" Oct 30 16:54:19.532595 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:19.532546 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce9ebd7f531e0576be41f3409c3c310dfb6899e8c4e20c02b0c2cde211e6205"} err="failed to get container status \"4ce9ebd7f531e0576be41f3409c3c310dfb6899e8c4e20c02b0c2cde211e6205\": rpc error: code = NotFound desc = could not find container \"4ce9ebd7f531e0576be41f3409c3c310dfb6899e8c4e20c02b0c2cde211e6205\": container with ID starting with 4ce9ebd7f531e0576be41f3409c3c310dfb6899e8c4e20c02b0c2cde211e6205 not found: ID does not exist" Oct 30 16:54:19.543643 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:54:19.543611 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-fp9rc\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="a084bdb7-f7be-4174-ac84-b3aba11e93a6" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-fp9rc" Oct 30 16:54:19.558386 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:54:19.558343 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-4pxbg\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="5d91990d-34b7-47eb-bcd0-ba3973a58615" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-4pxbg" Oct 30 16:54:20.487704 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:20.487669 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d91990d-34b7-47eb-bcd0-ba3973a58615" path="/var/lib/kubelet/pods/5d91990d-34b7-47eb-bcd0-ba3973a58615/volumes" Oct 30 16:54:20.488001 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:20.487989 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a084bdb7-f7be-4174-ac84-b3aba11e93a6" path="/var/lib/kubelet/pods/a084bdb7-f7be-4174-ac84-b3aba11e93a6/volumes" Oct 30 16:54:20.899633 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:20.899585 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:21.053963 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:21.053922 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:23.897601 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:23.897569 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:24.043352 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:24.043314 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:28.879259 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:28.879225 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:29.030656 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:29.030622 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:29.519635 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:29.519605 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" Oct 30 16:54:29.699506 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:29.699473 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j"] Oct 30 16:54:29.699723 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:29.699672 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" podUID="c3c4f0b8-5630-43bf-8a73-01a21547f55c" containerName="manager" containerID="cri-o://fd54a936e74b3a4a418f00601b788b86649ded874725508b77ea6fb46f458211" gracePeriod=10 Oct 30 16:54:29.701788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:29.701760 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" Oct 30 16:54:29.951316 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:29.951286 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" Oct 30 16:54:30.054219 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.054183 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3c4f0b8-5630-43bf-8a73-01a21547f55c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3c4f0b8-5630-43bf-8a73-01a21547f55c-extensions-socket-volume\") pod \"c3c4f0b8-5630-43bf-8a73-01a21547f55c\" (UID: \"c3c4f0b8-5630-43bf-8a73-01a21547f55c\") " Oct 30 16:54:30.054219 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.054225 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3c4f0b8-5630-43bf-8a73-01a21547f55c-kube-api-access-mzk6f\" (UniqueName: \"kubernetes.io/projected/c3c4f0b8-5630-43bf-8a73-01a21547f55c-kube-api-access-mzk6f\") pod \"c3c4f0b8-5630-43bf-8a73-01a21547f55c\" (UID: \"c3c4f0b8-5630-43bf-8a73-01a21547f55c\") " Oct 30 16:54:30.054624 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.054598 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3c4f0b8-5630-43bf-8a73-01a21547f55c-extensions-socket-volume" pod "c3c4f0b8-5630-43bf-8a73-01a21547f55c" (UID: "c3c4f0b8-5630-43bf-8a73-01a21547f55c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:54:30.056271 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.056248 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c4f0b8-5630-43bf-8a73-01a21547f55c-kube-api-access-mzk6f" pod "c3c4f0b8-5630-43bf-8a73-01a21547f55c" (UID: "c3c4f0b8-5630-43bf-8a73-01a21547f55c"). InnerVolumeSpecName "kube-api-access-mzk6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:54:30.084968 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.084865 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj"] Oct 30 16:54:30.085315 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.085298 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c3c4f0b8-5630-43bf-8a73-01a21547f55c" containerName="manager" Oct 30 16:54:30.085398 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.085318 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3c4f0b8-5630-43bf-8a73-01a21547f55c" containerName="manager" Oct 30 16:54:30.085451 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.085417 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3c4f0b8-5630-43bf-8a73-01a21547f55c" containerName="manager" Oct 30 16:54:30.088478 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.088456 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" Oct 30 16:54:30.121233 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.121199 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj"] Oct 30 16:54:30.155057 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.155017 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jr9c\" (UniqueName: \"kubernetes.io/projected/987bce29-ba9c-4a66-b285-f8706441e626-kube-api-access-4jr9c\") pod \"kuadrant-operator-controller-manager-57bb9bfff-n2gnj\" (UID: \"987bce29-ba9c-4a66-b285-f8706441e626\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" Oct 30 16:54:30.155057 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.155059 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/987bce29-ba9c-4a66-b285-f8706441e626-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-n2gnj\" (UID: \"987bce29-ba9c-4a66-b285-f8706441e626\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" Oct 30 16:54:30.155248 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.155185 4828 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c3c4f0b8-5630-43bf-8a73-01a21547f55c-extensions-socket-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:54:30.155248 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.155197 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzk6f\" (UniqueName: \"kubernetes.io/projected/c3c4f0b8-5630-43bf-8a73-01a21547f55c-kube-api-access-mzk6f\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:54:30.256116 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.256083 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jr9c\" (UniqueName: \"kubernetes.io/projected/987bce29-ba9c-4a66-b285-f8706441e626-kube-api-access-4jr9c\") pod \"kuadrant-operator-controller-manager-57bb9bfff-n2gnj\" (UID: \"987bce29-ba9c-4a66-b285-f8706441e626\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" Oct 30 16:54:30.256285 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.256133 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/987bce29-ba9c-4a66-b285-f8706441e626-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-n2gnj\" (UID: \"987bce29-ba9c-4a66-b285-f8706441e626\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" Oct 30 16:54:30.256521 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.256504 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/987bce29-ba9c-4a66-b285-f8706441e626-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-n2gnj\" (UID: \"987bce29-ba9c-4a66-b285-f8706441e626\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" Oct 30 16:54:30.286170 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.286136 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jr9c\" (UniqueName: \"kubernetes.io/projected/987bce29-ba9c-4a66-b285-f8706441e626-kube-api-access-4jr9c\") pod \"kuadrant-operator-controller-manager-57bb9bfff-n2gnj\" (UID: \"987bce29-ba9c-4a66-b285-f8706441e626\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" Oct 30 16:54:30.401568 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.401468 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" Oct 30 16:54:30.577437 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.577391 4828 generic.go:356] "Generic (PLEG): container finished" podID="c3c4f0b8-5630-43bf-8a73-01a21547f55c" containerID="fd54a936e74b3a4a418f00601b788b86649ded874725508b77ea6fb46f458211" exitCode=0 Oct 30 16:54:30.577607 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.577448 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" Oct 30 16:54:30.577607 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.577479 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" event={"ID":"c3c4f0b8-5630-43bf-8a73-01a21547f55c","Type":"ContainerDied","Data":"fd54a936e74b3a4a418f00601b788b86649ded874725508b77ea6fb46f458211"} Oct 30 16:54:30.577607 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.577539 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j" event={"ID":"c3c4f0b8-5630-43bf-8a73-01a21547f55c","Type":"ContainerDied","Data":"1f8b3ca98d15014637cbb53262914648d1feeac568048f6ae61f55a733865ad5"} Oct 30 16:54:30.577607 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.577563 4828 scope.go:122] "RemoveContainer" containerID="fd54a936e74b3a4a418f00601b788b86649ded874725508b77ea6fb46f458211" Oct 30 16:54:30.586302 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.586275 4828 scope.go:122] "RemoveContainer" containerID="fd54a936e74b3a4a418f00601b788b86649ded874725508b77ea6fb46f458211" Oct 30 16:54:30.586638 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:54:30.586613 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd54a936e74b3a4a418f00601b788b86649ded874725508b77ea6fb46f458211\": container with ID starting with fd54a936e74b3a4a418f00601b788b86649ded874725508b77ea6fb46f458211 not found: ID does not exist" containerID="fd54a936e74b3a4a418f00601b788b86649ded874725508b77ea6fb46f458211" Oct 30 16:54:30.586716 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.586648 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd54a936e74b3a4a418f00601b788b86649ded874725508b77ea6fb46f458211"} err="failed to get container status \"fd54a936e74b3a4a418f00601b788b86649ded874725508b77ea6fb46f458211\": rpc error: code = NotFound desc = could not find container \"fd54a936e74b3a4a418f00601b788b86649ded874725508b77ea6fb46f458211\": container with ID starting with fd54a936e74b3a4a418f00601b788b86649ded874725508b77ea6fb46f458211 not found: ID does not exist" Oct 30 16:54:30.635237 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.635198 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j"] Oct 30 16:54:30.655102 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.655018 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-wm96j"] Oct 30 16:54:30.717253 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:54:30.717220 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987bce29_ba9c_4a66_b285_f8706441e626.slice/crio-c73db37b580bdf7ab8d4b09a9e810ac6d97cad630441f8eba423a52ec1bfa7e5 WatchSource:0}: Error finding container c73db37b580bdf7ab8d4b09a9e810ac6d97cad630441f8eba423a52ec1bfa7e5: Status 404 returned error can't find the container with id c73db37b580bdf7ab8d4b09a9e810ac6d97cad630441f8eba423a52ec1bfa7e5 Oct 30 16:54:30.717371 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:30.717276 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj"] Oct 30 16:54:31.585519 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:31.585474 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" event={"ID":"987bce29-ba9c-4a66-b285-f8706441e626","Type":"ContainerStarted","Data":"0a46fc60812b679ae1e88c81173c005e6f0da49df6a530055360c5958661fb8f"} Oct 30 16:54:31.585959 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:31.585545 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" event={"ID":"987bce29-ba9c-4a66-b285-f8706441e626","Type":"ContainerStarted","Data":"c73db37b580bdf7ab8d4b09a9e810ac6d97cad630441f8eba423a52ec1bfa7e5"} Oct 30 16:54:31.585959 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:31.585574 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" Oct 30 16:54:31.660830 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:31.660710 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" podStartSLOduration=1.660690138 podStartE2EDuration="1.660690138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:31.646377415 +0000 UTC m=+449.291887594" watchObservedRunningTime="2026-10-30 16:54:31.660690138 +0000 UTC m=+449.306200320" Oct 30 16:54:32.488035 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:32.488002 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3c4f0b8-5630-43bf-8a73-01a21547f55c" path="/var/lib/kubelet/pods/c3c4f0b8-5630-43bf-8a73-01a21547f55c/volumes" Oct 30 16:54:35.395072 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:35.395033 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:35.553066 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:35.553021 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:37.910364 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:54:37.910326 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:54:42.594242 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:42.594210 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" Oct 30 16:54:42.692776 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:42.692747 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n"] Oct 30 16:54:42.693274 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:42.693244 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" podUID="5238d15a-6a60-4e03-bb72-9daec5a28e9b" containerName="manager" containerID="cri-o://7632ccbb649923c7c330e3853a38ed5543f2f98836a43c04cc69aa0cf1744aa0" gracePeriod=10 Oct 30 16:54:42.890280 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:42.890244 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:42.943842 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:42.943816 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" Oct 30 16:54:43.047334 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.047297 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:43.070283 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.070246 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5238d15a-6a60-4e03-bb72-9daec5a28e9b-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5238d15a-6a60-4e03-bb72-9daec5a28e9b-extensions-socket-volume\") pod \"5238d15a-6a60-4e03-bb72-9daec5a28e9b\" (UID: \"5238d15a-6a60-4e03-bb72-9daec5a28e9b\") " Oct 30 16:54:43.070456 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.070340 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5238d15a-6a60-4e03-bb72-9daec5a28e9b-kube-api-access-8x9vc\" (UniqueName: \"kubernetes.io/projected/5238d15a-6a60-4e03-bb72-9daec5a28e9b-kube-api-access-8x9vc\") pod \"5238d15a-6a60-4e03-bb72-9daec5a28e9b\" (UID: \"5238d15a-6a60-4e03-bb72-9daec5a28e9b\") " Oct 30 16:54:43.070666 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.070639 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5238d15a-6a60-4e03-bb72-9daec5a28e9b-extensions-socket-volume" pod "5238d15a-6a60-4e03-bb72-9daec5a28e9b" (UID: "5238d15a-6a60-4e03-bb72-9daec5a28e9b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:54:43.072323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.072304 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5238d15a-6a60-4e03-bb72-9daec5a28e9b-kube-api-access-8x9vc" pod "5238d15a-6a60-4e03-bb72-9daec5a28e9b" (UID: "5238d15a-6a60-4e03-bb72-9daec5a28e9b"). InnerVolumeSpecName "kube-api-access-8x9vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:54:43.171566 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.171483 4828 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5238d15a-6a60-4e03-bb72-9daec5a28e9b-extensions-socket-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:54:43.171566 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.171512 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x9vc\" (UniqueName: \"kubernetes.io/projected/5238d15a-6a60-4e03-bb72-9daec5a28e9b-kube-api-access-8x9vc\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:54:43.493093 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.493006 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:43.652688 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.652651 4828 generic.go:356] "Generic (PLEG): container finished" podID="5238d15a-6a60-4e03-bb72-9daec5a28e9b" containerID="7632ccbb649923c7c330e3853a38ed5543f2f98836a43c04cc69aa0cf1744aa0" exitCode=0 Oct 30 16:54:43.652688 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.652690 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" event={"ID":"5238d15a-6a60-4e03-bb72-9daec5a28e9b","Type":"ContainerDied","Data":"7632ccbb649923c7c330e3853a38ed5543f2f98836a43c04cc69aa0cf1744aa0"} Oct 30 16:54:43.653203 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.652723 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" Oct 30 16:54:43.653203 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.652752 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n" event={"ID":"5238d15a-6a60-4e03-bb72-9daec5a28e9b","Type":"ContainerDied","Data":"9a1cc0f16b828cfda5fc63ed4ebafd05f5ea3109cb403903bb3cda3d02d5139f"} Oct 30 16:54:43.653203 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.652773 4828 scope.go:122] "RemoveContainer" containerID="7632ccbb649923c7c330e3853a38ed5543f2f98836a43c04cc69aa0cf1744aa0" Oct 30 16:54:43.655665 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.655639 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:43.663301 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.663280 4828 scope.go:122] "RemoveContainer" containerID="7632ccbb649923c7c330e3853a38ed5543f2f98836a43c04cc69aa0cf1744aa0" Oct 30 16:54:43.663592 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:54:43.663573 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7632ccbb649923c7c330e3853a38ed5543f2f98836a43c04cc69aa0cf1744aa0\": container with ID starting with 7632ccbb649923c7c330e3853a38ed5543f2f98836a43c04cc69aa0cf1744aa0 not found: ID does not exist" containerID="7632ccbb649923c7c330e3853a38ed5543f2f98836a43c04cc69aa0cf1744aa0" Oct 30 16:54:43.663637 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.663601 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7632ccbb649923c7c330e3853a38ed5543f2f98836a43c04cc69aa0cf1744aa0"} err="failed to get container status \"7632ccbb649923c7c330e3853a38ed5543f2f98836a43c04cc69aa0cf1744aa0\": rpc error: code = NotFound desc = could not find container \"7632ccbb649923c7c330e3853a38ed5543f2f98836a43c04cc69aa0cf1744aa0\": container with ID starting with 7632ccbb649923c7c330e3853a38ed5543f2f98836a43c04cc69aa0cf1744aa0 not found: ID does not exist" Oct 30 16:54:43.725424 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.725388 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n"] Oct 30 16:54:43.739022 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.738982 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-vtv9n"] Oct 30 16:54:43.830985 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.830950 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:43.988890 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:43.988858 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:44.487477 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:44.487443 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5238d15a-6a60-4e03-bb72-9daec5a28e9b" path="/var/lib/kubelet/pods/5238d15a-6a60-4e03-bb72-9daec5a28e9b/volumes" Oct 30 16:54:47.015461 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.015414 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb"] Oct 30 16:54:47.016050 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.015950 4828 cpu_manager.go:414] "containerMap: removing container" podUID="5238d15a-6a60-4e03-bb72-9daec5a28e9b" containerName="manager" Oct 30 16:54:47.016050 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.015970 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5238d15a-6a60-4e03-bb72-9daec5a28e9b" containerName="manager" Oct 30 16:54:47.016165 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.016087 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="5238d15a-6a60-4e03-bb72-9daec5a28e9b" containerName="manager" Oct 30 16:54:47.018362 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.018336 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.044134 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.044109 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fcls9\"" Oct 30 16:54:47.056975 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.056942 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb"] Oct 30 16:54:47.107235 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.107201 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.107235 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.107239 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.107458 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.107258 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.107458 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.107282 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.107458 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.107404 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.107458 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.107429 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.107458 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.107452 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.107706 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.107468 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr9xq\" (UniqueName: \"kubernetes.io/projected/1fc9a9e7-416b-4b87-912e-a524f41e0d53-kube-api-access-wr9xq\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.107706 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.107636 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.114268 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.114236 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:47.208465 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.208424 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.208659 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.208469 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.208659 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.208495 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.208659 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.208530 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.208659 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.208567 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.208659 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.208591 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.208659 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.208619 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.208659 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.208643 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr9xq\" (UniqueName: \"kubernetes.io/projected/1fc9a9e7-416b-4b87-912e-a524f41e0d53-kube-api-access-wr9xq\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.209051 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.208733 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.209051 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.209009 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.209164 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.209102 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.209164 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.209140 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.209407 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.209382 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.209490 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.209463 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.210828 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.210806 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.210996 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.210977 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.233313 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.233280 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fc9a9e7-416b-4b87-912e-a524f41e0d53-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.233484 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.233347 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr9xq\" (UniqueName: \"kubernetes.io/projected/1fc9a9e7-416b-4b87-912e-a524f41e0d53-kube-api-access-wr9xq\") pod \"maas-default-gateway-openshift-default-6db586dd6c-r26lb\" (UID: \"1fc9a9e7-416b-4b87-912e-a524f41e0d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.288520 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.288415 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:47.331558 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.331527 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:47.443331 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.443295 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:47.646722 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.646697 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb"] Oct 30 16:54:47.648446 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:54:47.648412 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc9a9e7_416b_4b87_912e_a524f41e0d53.slice/crio-bab5e7b2be721e3e7edefbe5e9c2e243bb0aaa55050b34f83567fc5e46494e0a WatchSource:0}: Error finding container bab5e7b2be721e3e7edefbe5e9c2e243bb0aaa55050b34f83567fc5e46494e0a: Status 404 returned error can't find the container with id bab5e7b2be721e3e7edefbe5e9c2e243bb0aaa55050b34f83567fc5e46494e0a Oct 30 16:54:47.675091 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:47.675048 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" event={"ID":"1fc9a9e7-416b-4b87-912e-a524f41e0d53","Type":"ContainerStarted","Data":"bab5e7b2be721e3e7edefbe5e9c2e243bb0aaa55050b34f83567fc5e46494e0a"} Oct 30 16:54:50.062019 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:50.061983 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:54:50.062406 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:50.062070 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:54:50.062406 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:50.062115 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:54:50.694352 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:50.694313 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" event={"ID":"1fc9a9e7-416b-4b87-912e-a524f41e0d53","Type":"ContainerStarted","Data":"5ae5a187df95bc23f1d80be94a3b53ae9f4a9c96ac5d71102467d639a3049f42"} Oct 30 16:54:50.758677 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:50.758621 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" podStartSLOduration=2.347946587 podStartE2EDuration="4.758608402s" totalImagesPullingTime="2.410661815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:50.758417782 +0000 UTC m=+468.403927962" watchObservedRunningTime="2026-10-30 16:54:50.758608402 +0000 UTC m=+468.404118580" Oct 30 16:54:51.331998 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:51.331954 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:51.336973 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:51.336948 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:51.699214 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:51.699127 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:51.700131 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:51.700109 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:54:53.225722 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:53.225678 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:53.376735 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:53.376698 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:53.885246 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:53.885203 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:54.040147 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:54.040115 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:54.485758 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:54.485721 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:54:54.636085 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:54:54.636052 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:01.082478 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:01.082440 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:01.247966 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:01.247936 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:02.656878 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.656833 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zqmvk"] Oct 30 16:55:02.660622 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.660598 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" Oct 30 16:55:02.680696 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.680664 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zqmvk"] Oct 30 16:55:02.684386 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.684348 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:02.691831 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.691801 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 16:55:02.691831 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.691842 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wt7tg\"" Oct 30 16:55:02.746720 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.746686 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msm6k\" (UniqueName: \"kubernetes.io/projected/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-kube-api-access-msm6k\") pod \"limitador-limitador-74dfcfb894-zqmvk\" (UID: \"c3020e9f-30b2-4456-851a-1fbe7e9ae06b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" Oct 30 16:55:02.746887 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.746807 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-config-file\") pod \"limitador-limitador-74dfcfb894-zqmvk\" (UID: \"c3020e9f-30b2-4456-851a-1fbe7e9ae06b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" Oct 30 16:55:02.800962 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.800862 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zqmvk"] Oct 30 16:55:02.847760 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.847724 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-config-file\") pod \"limitador-limitador-74dfcfb894-zqmvk\" (UID: \"c3020e9f-30b2-4456-851a-1fbe7e9ae06b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" Oct 30 16:55:02.847977 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.847783 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msm6k\" (UniqueName: \"kubernetes.io/projected/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-kube-api-access-msm6k\") pod \"limitador-limitador-74dfcfb894-zqmvk\" (UID: \"c3020e9f-30b2-4456-851a-1fbe7e9ae06b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" Oct 30 16:55:02.848509 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.848487 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-config-file\") pod \"limitador-limitador-74dfcfb894-zqmvk\" (UID: \"c3020e9f-30b2-4456-851a-1fbe7e9ae06b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" Oct 30 16:55:02.872836 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.872801 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msm6k\" (UniqueName: \"kubernetes.io/projected/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-kube-api-access-msm6k\") pod \"limitador-limitador-74dfcfb894-zqmvk\" (UID: \"c3020e9f-30b2-4456-851a-1fbe7e9ae06b\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" Oct 30 16:55:02.909233 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.909198 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:02.972396 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:02.972364 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" Oct 30 16:55:03.057521 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:03.057434 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:03.273463 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:03.273324 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zqmvk"] Oct 30 16:55:03.276006 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:55:03.275972 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3020e9f_30b2_4456_851a_1fbe7e9ae06b.slice/crio-2cce8ffdd93be71822cd2899f825b7a85ce6a547e17a3ae1b463338dce486b55 WatchSource:0}: Error finding container 2cce8ffdd93be71822cd2899f825b7a85ce6a547e17a3ae1b463338dce486b55: Status 404 returned error can't find the container with id 2cce8ffdd93be71822cd2899f825b7a85ce6a547e17a3ae1b463338dce486b55 Oct 30 16:55:03.745955 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:03.745919 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:03.758971 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:03.758910 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" event={"ID":"c3020e9f-30b2-4456-851a-1fbe7e9ae06b","Type":"ContainerStarted","Data":"2cce8ffdd93be71822cd2899f825b7a85ce6a547e17a3ae1b463338dce486b55"} Oct 30 16:55:03.899589 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:03.899552 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:05.770275 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:05.770233 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" event={"ID":"c3020e9f-30b2-4456-851a-1fbe7e9ae06b","Type":"ContainerStarted","Data":"0c4a685a088a3784f6a9d8fde14a4984ced0d6f3cd1e2484440a02e9482705c0"} Oct 30 16:55:05.770713 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:05.770313 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" Oct 30 16:55:05.816199 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:05.816144 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" podStartSLOduration=2.324283752 podStartE2EDuration="3.816129895s" totalImagesPullingTime="1.491846143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:05.814632918 +0000 UTC m=+483.460143099" watchObservedRunningTime="2026-10-30 16:55:05.816129895 +0000 UTC m=+483.461640075" Oct 30 16:55:06.896757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:06.896725 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:07.047126 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:07.047093 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:07.910207 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:55:07.910168 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:55:09.906489 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:09.906430 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:10.050879 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:10.050840 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:16.776659 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:16.776584 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" Oct 30 16:55:16.875594 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:16.875551 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zqmvk"] Oct 30 16:55:16.875806 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:16.875763 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" podUID="c3020e9f-30b2-4456-851a-1fbe7e9ae06b" containerName="limitador" containerID="cri-o://0c4a685a088a3784f6a9d8fde14a4984ced0d6f3cd1e2484440a02e9482705c0" gracePeriod=30 Oct 30 16:55:17.429388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.429361 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" Oct 30 16:55:17.466111 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.466033 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-config-file\" (UniqueName: \"kubernetes.io/configmap/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-config-file\") pod \"c3020e9f-30b2-4456-851a-1fbe7e9ae06b\" (UID: \"c3020e9f-30b2-4456-851a-1fbe7e9ae06b\") " Oct 30 16:55:17.466420 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.466398 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-config-file" pod "c3020e9f-30b2-4456-851a-1fbe7e9ae06b" (UID: "c3020e9f-30b2-4456-851a-1fbe7e9ae06b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:55:17.566600 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.566566 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-kube-api-access-msm6k\" (UniqueName: \"kubernetes.io/projected/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-kube-api-access-msm6k\") pod \"c3020e9f-30b2-4456-851a-1fbe7e9ae06b\" (UID: \"c3020e9f-30b2-4456-851a-1fbe7e9ae06b\") " Oct 30 16:55:17.566924 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.566872 4828 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-config-file\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:55:17.568532 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.568511 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-kube-api-access-msm6k" pod "c3020e9f-30b2-4456-851a-1fbe7e9ae06b" (UID: "c3020e9f-30b2-4456-851a-1fbe7e9ae06b"). InnerVolumeSpecName "kube-api-access-msm6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:17.667668 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.667631 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msm6k\" (UniqueName: \"kubernetes.io/projected/c3020e9f-30b2-4456-851a-1fbe7e9ae06b-kube-api-access-msm6k\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:55:17.831820 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.831785 4828 generic.go:356] "Generic (PLEG): container finished" podID="c3020e9f-30b2-4456-851a-1fbe7e9ae06b" containerID="0c4a685a088a3784f6a9d8fde14a4984ced0d6f3cd1e2484440a02e9482705c0" exitCode=0 Oct 30 16:55:17.832318 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.831827 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" Oct 30 16:55:17.832318 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.831847 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" event={"ID":"c3020e9f-30b2-4456-851a-1fbe7e9ae06b","Type":"ContainerDied","Data":"0c4a685a088a3784f6a9d8fde14a4984ced0d6f3cd1e2484440a02e9482705c0"} Oct 30 16:55:17.832318 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.831888 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zqmvk" event={"ID":"c3020e9f-30b2-4456-851a-1fbe7e9ae06b","Type":"ContainerDied","Data":"2cce8ffdd93be71822cd2899f825b7a85ce6a547e17a3ae1b463338dce486b55"} Oct 30 16:55:17.832318 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.831937 4828 scope.go:122] "RemoveContainer" containerID="0c4a685a088a3784f6a9d8fde14a4984ced0d6f3cd1e2484440a02e9482705c0" Oct 30 16:55:17.841736 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.841718 4828 scope.go:122] "RemoveContainer" containerID="0c4a685a088a3784f6a9d8fde14a4984ced0d6f3cd1e2484440a02e9482705c0" Oct 30 16:55:17.842059 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:55:17.842040 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4a685a088a3784f6a9d8fde14a4984ced0d6f3cd1e2484440a02e9482705c0\": container with ID starting with 0c4a685a088a3784f6a9d8fde14a4984ced0d6f3cd1e2484440a02e9482705c0 not found: ID does not exist" containerID="0c4a685a088a3784f6a9d8fde14a4984ced0d6f3cd1e2484440a02e9482705c0" Oct 30 16:55:17.842118 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.842072 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4a685a088a3784f6a9d8fde14a4984ced0d6f3cd1e2484440a02e9482705c0"} err="failed to get container status \"0c4a685a088a3784f6a9d8fde14a4984ced0d6f3cd1e2484440a02e9482705c0\": rpc error: code = NotFound desc = could not find container \"0c4a685a088a3784f6a9d8fde14a4984ced0d6f3cd1e2484440a02e9482705c0\": container with ID starting with 0c4a685a088a3784f6a9d8fde14a4984ced0d6f3cd1e2484440a02e9482705c0 not found: ID does not exist" Oct 30 16:55:17.888484 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.888438 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zqmvk"] Oct 30 16:55:17.901929 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:17.901885 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zqmvk"] Oct 30 16:55:18.488561 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:18.488527 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3020e9f-30b2-4456-851a-1fbe7e9ae06b" path="/var/lib/kubelet/pods/c3020e9f-30b2-4456-851a-1fbe7e9ae06b/volumes" Oct 30 16:55:19.306886 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:19.306844 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:19.487576 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:19.487543 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:20.031714 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.031677 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs"] Oct 30 16:55:20.032073 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.032061 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c3020e9f-30b2-4456-851a-1fbe7e9ae06b" containerName="limitador" Oct 30 16:55:20.032134 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.032074 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3020e9f-30b2-4456-851a-1fbe7e9ae06b" containerName="limitador" Oct 30 16:55:20.032169 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.032133 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3020e9f-30b2-4456-851a-1fbe7e9ae06b" containerName="limitador" Oct 30 16:55:20.038085 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.038067 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs" Oct 30 16:55:20.056741 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.056686 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs"] Oct 30 16:55:20.057066 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.057041 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:55:20.068273 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.068236 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:55:20.068273 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.068257 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 16:55:20.068509 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.068302 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9k99s\"" Oct 30 16:55:20.082365 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.082326 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d1396d19-0a39-4d00-bbf0-ce5cd260354c-data\") pod \"postgres-7cdbf6dc44-c9mfs\" (UID: \"d1396d19-0a39-4d00-bbf0-ce5cd260354c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs" Oct 30 16:55:20.082560 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.082510 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcjps\" (UniqueName: \"kubernetes.io/projected/d1396d19-0a39-4d00-bbf0-ce5cd260354c-kube-api-access-xcjps\") pod \"postgres-7cdbf6dc44-c9mfs\" (UID: \"d1396d19-0a39-4d00-bbf0-ce5cd260354c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs" Oct 30 16:55:20.165971 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.165932 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:20.183248 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.183179 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcjps\" (UniqueName: \"kubernetes.io/projected/d1396d19-0a39-4d00-bbf0-ce5cd260354c-kube-api-access-xcjps\") pod \"postgres-7cdbf6dc44-c9mfs\" (UID: \"d1396d19-0a39-4d00-bbf0-ce5cd260354c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs" Oct 30 16:55:20.183632 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.183267 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d1396d19-0a39-4d00-bbf0-ce5cd260354c-data\") pod \"postgres-7cdbf6dc44-c9mfs\" (UID: \"d1396d19-0a39-4d00-bbf0-ce5cd260354c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs" Oct 30 16:55:20.183868 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.183847 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d1396d19-0a39-4d00-bbf0-ce5cd260354c-data\") pod \"postgres-7cdbf6dc44-c9mfs\" (UID: \"d1396d19-0a39-4d00-bbf0-ce5cd260354c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs" Oct 30 16:55:20.208866 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.208834 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcjps\" (UniqueName: \"kubernetes.io/projected/d1396d19-0a39-4d00-bbf0-ce5cd260354c-kube-api-access-xcjps\") pod \"postgres-7cdbf6dc44-c9mfs\" (UID: \"d1396d19-0a39-4d00-bbf0-ce5cd260354c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs" Oct 30 16:55:20.353799 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.353713 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs" Oct 30 16:55:20.655799 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.655771 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs"] Oct 30 16:55:20.657014 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:55:20.656988 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1396d19_0a39_4d00_bbf0_ce5cd260354c.slice/crio-a682cd07dce77f8fefc09eb359be3d136ad1c96c835128f1357926f973972c40 WatchSource:0}: Error finding container a682cd07dce77f8fefc09eb359be3d136ad1c96c835128f1357926f973972c40: Status 404 returned error can't find the container with id a682cd07dce77f8fefc09eb359be3d136ad1c96c835128f1357926f973972c40 Oct 30 16:55:20.853632 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:20.853593 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs" event={"ID":"d1396d19-0a39-4d00-bbf0-ce5cd260354c","Type":"ContainerStarted","Data":"a682cd07dce77f8fefc09eb359be3d136ad1c96c835128f1357926f973972c40"} Oct 30 16:55:25.887641 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:25.887546 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs" event={"ID":"d1396d19-0a39-4d00-bbf0-ce5cd260354c","Type":"ContainerStarted","Data":"27623991471e61d6ef8202f9dc987e2447d40b436550ea30ff3b844558d92423"} Oct 30 16:55:25.888085 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:25.887732 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs" Oct 30 16:55:25.933280 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:25.933226 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs" podStartSLOduration=1.002595827 podStartE2EDuration="5.933211526s" totalImagesPullingTime="4.930615699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:25.931677691 +0000 UTC m=+503.577187870" watchObservedRunningTime="2026-10-30 16:55:25.933211526 +0000 UTC m=+503.578721727" Oct 30 16:55:31.927645 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:31.927617 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-c9mfs" Oct 30 16:55:32.479029 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:32.478988 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:32.674269 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:32.674229 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:37.909796 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:55:37.909759 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:55:39.087425 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:39.087388 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:42.903321 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:42.903272 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:42.906528 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:42.906501 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-qg4fp"] Oct 30 16:55:42.912061 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:42.912040 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:42.928258 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:42.928225 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 16:55:42.928258 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:42.928245 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 16:55:42.928425 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:42.928269 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8xm5f\"" Oct 30 16:55:42.928425 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:42.928301 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 16:55:42.943709 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:42.943674 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-qg4fp"] Oct 30 16:55:42.946625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:42.946602 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-webhook-cert\") pod \"maas-controller-577f596567-qg4fp\" (UID: \"e043f5d8-e5d2-4f87-9797-f36165152350\") " pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:42.946715 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:42.946660 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbvv\" (UniqueName: \"kubernetes.io/projected/e043f5d8-e5d2-4f87-9797-f36165152350-kube-api-access-lmbvv\") pod \"maas-controller-577f596567-qg4fp\" (UID: \"e043f5d8-e5d2-4f87-9797-f36165152350\") " pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:42.946780 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:42.946765 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-metrics-certs\") pod \"maas-controller-577f596567-qg4fp\" (UID: \"e043f5d8-e5d2-4f87-9797-f36165152350\") " pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:43.048244 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.048175 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-metrics-certs\") pod \"maas-controller-577f596567-qg4fp\" (UID: \"e043f5d8-e5d2-4f87-9797-f36165152350\") " pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:43.048437 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.048263 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-webhook-cert\") pod \"maas-controller-577f596567-qg4fp\" (UID: \"e043f5d8-e5d2-4f87-9797-f36165152350\") " pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:43.048437 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.048329 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbvv\" (UniqueName: \"kubernetes.io/projected/e043f5d8-e5d2-4f87-9797-f36165152350-kube-api-access-lmbvv\") pod \"maas-controller-577f596567-qg4fp\" (UID: \"e043f5d8-e5d2-4f87-9797-f36165152350\") " pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:43.051751 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.051675 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-metrics-certs\") pod \"maas-controller-577f596567-qg4fp\" (UID: \"e043f5d8-e5d2-4f87-9797-f36165152350\") " pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:43.051751 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.051678 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-webhook-cert\") pod \"maas-controller-577f596567-qg4fp\" (UID: \"e043f5d8-e5d2-4f87-9797-f36165152350\") " pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:43.058793 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.058748 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:43.074240 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.074212 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbvv\" (UniqueName: \"kubernetes.io/projected/e043f5d8-e5d2-4f87-9797-f36165152350-kube-api-access-lmbvv\") pod \"maas-controller-577f596567-qg4fp\" (UID: \"e043f5d8-e5d2-4f87-9797-f36165152350\") " pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:43.211352 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.211260 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:43.225201 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.225164 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:43.317743 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.317705 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78487d6d64-hdjk9"] Oct 30 16:55:43.322693 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.322665 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 16:55:43.344132 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.344097 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78487d6d64-hdjk9"] Oct 30 16:55:43.351210 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.351173 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-metrics-certs\") pod \"maas-controller-78487d6d64-hdjk9\" (UID: \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\") " pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 16:55:43.351380 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.351339 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd7b7\" (UniqueName: \"kubernetes.io/projected/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-kube-api-access-kd7b7\") pod \"maas-controller-78487d6d64-hdjk9\" (UID: \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\") " pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 16:55:43.351444 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.351404 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-webhook-cert\") pod \"maas-controller-78487d6d64-hdjk9\" (UID: \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\") " pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 16:55:43.368043 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.367843 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:43.452216 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.452180 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd7b7\" (UniqueName: \"kubernetes.io/projected/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-kube-api-access-kd7b7\") pod \"maas-controller-78487d6d64-hdjk9\" (UID: \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\") " pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 16:55:43.452431 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.452240 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-webhook-cert\") pod \"maas-controller-78487d6d64-hdjk9\" (UID: \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\") " pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 16:55:43.452431 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.452267 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-metrics-certs\") pod \"maas-controller-78487d6d64-hdjk9\" (UID: \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\") " pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 16:55:43.454961 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.454933 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-metrics-certs\") pod \"maas-controller-78487d6d64-hdjk9\" (UID: \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\") " pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 16:55:43.454961 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.454953 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-webhook-cert\") pod \"maas-controller-78487d6d64-hdjk9\" (UID: \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\") " pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 16:55:43.479487 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.479416 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd7b7\" (UniqueName: \"kubernetes.io/projected/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-kube-api-access-kd7b7\") pod \"maas-controller-78487d6d64-hdjk9\" (UID: \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\") " pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 16:55:43.526287 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.526257 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-qg4fp"] Oct 30 16:55:43.526895 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:55:43.526868 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode043f5d8_e5d2_4f87_9797_f36165152350.slice/crio-46b9d308704dae1038e679f1398e7f4159d97e65f11bd008badaf65714b9b67c WatchSource:0}: Error finding container 46b9d308704dae1038e679f1398e7f4159d97e65f11bd008badaf65714b9b67c: Status 404 returned error can't find the container with id 46b9d308704dae1038e679f1398e7f4159d97e65f11bd008badaf65714b9b67c Oct 30 16:55:43.636553 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.636514 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 16:55:43.938574 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.938545 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78487d6d64-hdjk9"] Oct 30 16:55:43.939520 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:55:43.939495 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a0571b_6e0f_4fa9_bbd3_d0b31614b19b.slice/crio-dbf78ee80e0f4773ef61112c2c0b8d7f8b2658ad1c7f580c11a0f86828240d7c WatchSource:0}: Error finding container dbf78ee80e0f4773ef61112c2c0b8d7f8b2658ad1c7f580c11a0f86828240d7c: Status 404 returned error can't find the container with id dbf78ee80e0f4773ef61112c2c0b8d7f8b2658ad1c7f580c11a0f86828240d7c Oct 30 16:55:43.981058 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.981022 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-qg4fp" event={"ID":"e043f5d8-e5d2-4f87-9797-f36165152350","Type":"ContainerStarted","Data":"46b9d308704dae1038e679f1398e7f4159d97e65f11bd008badaf65714b9b67c"} Oct 30 16:55:43.982197 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:43.982169 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78487d6d64-hdjk9" event={"ID":"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b","Type":"ContainerStarted","Data":"dbf78ee80e0f4773ef61112c2c0b8d7f8b2658ad1c7f580c11a0f86828240d7c"} Oct 30 16:55:46.624130 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:46.624098 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:46.782020 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:46.781986 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:47.003001 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:47.002884 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-qg4fp" event={"ID":"e043f5d8-e5d2-4f87-9797-f36165152350","Type":"ContainerStarted","Data":"f49cff2ac15f4857fad0f9bd93c669a50caf0fde1ed0639bdbcbda3b20bf0a17"} Oct 30 16:55:47.003001 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:47.002971 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:47.004406 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:47.004381 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78487d6d64-hdjk9" event={"ID":"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b","Type":"ContainerStarted","Data":"79a08b1bd537066ad655e5be4faa997c7746d8b67baa818b8e044ae0c0f50b51"} Oct 30 16:55:47.004527 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:47.004460 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 16:55:47.071710 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:47.071648 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-qg4fp" podStartSLOduration=2.293863257 podStartE2EDuration="5.071630189s" totalImagesPullingTime="2.777766932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:47.069244593 +0000 UTC m=+524.714754774" watchObservedRunningTime="2026-10-30 16:55:47.071630189 +0000 UTC m=+524.717140372" Oct 30 16:55:47.121530 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:47.121472 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78487d6d64-hdjk9" podStartSLOduration=1.502027221 podStartE2EDuration="4.121455526s" totalImagesPullingTime="2.619428305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:47.119335961 +0000 UTC m=+524.764846140" watchObservedRunningTime="2026-10-30 16:55:47.121455526 +0000 UTC m=+524.766965743" Oct 30 16:55:50.141519 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.141482 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs"] Oct 30 16:55:50.145196 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.145172 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:50.176763 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.176733 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 16:55:50.177388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.177366 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 16:55:50.177716 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.177697 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 16:55:50.178927 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.178038 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-n46pw\"" Oct 30 16:55:50.179849 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.179826 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs"] Oct 30 16:55:50.215197 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.215167 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/177c030c-d783-44ae-92e9-3368ed00ef0d-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b8bbs\" (UID: \"177c030c-d783-44ae-92e9-3368ed00ef0d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:50.215362 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.215230 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrv58\" (UniqueName: \"kubernetes.io/projected/177c030c-d783-44ae-92e9-3368ed00ef0d-kube-api-access-wrv58\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b8bbs\" (UID: \"177c030c-d783-44ae-92e9-3368ed00ef0d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:50.215362 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.215308 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/177c030c-d783-44ae-92e9-3368ed00ef0d-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b8bbs\" (UID: \"177c030c-d783-44ae-92e9-3368ed00ef0d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:50.215362 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.215334 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/177c030c-d783-44ae-92e9-3368ed00ef0d-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b8bbs\" (UID: \"177c030c-d783-44ae-92e9-3368ed00ef0d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:50.267805 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.267468 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:50.315825 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.315786 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrv58\" (UniqueName: \"kubernetes.io/projected/177c030c-d783-44ae-92e9-3368ed00ef0d-kube-api-access-wrv58\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b8bbs\" (UID: \"177c030c-d783-44ae-92e9-3368ed00ef0d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:50.316061 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.315832 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/177c030c-d783-44ae-92e9-3368ed00ef0d-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b8bbs\" (UID: \"177c030c-d783-44ae-92e9-3368ed00ef0d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:50.316061 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.315851 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/177c030c-d783-44ae-92e9-3368ed00ef0d-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b8bbs\" (UID: \"177c030c-d783-44ae-92e9-3368ed00ef0d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:50.316061 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.315893 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/177c030c-d783-44ae-92e9-3368ed00ef0d-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b8bbs\" (UID: \"177c030c-d783-44ae-92e9-3368ed00ef0d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:50.316061 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:55:50.316009 4828 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 16:55:50.316246 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:55:50.316083 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/177c030c-d783-44ae-92e9-3368ed00ef0d-tls-cert podName:177c030c-d783-44ae-92e9-3368ed00ef0d nodeName:}" failed. No retries permitted until 2026-10-30 16:55:50.816066826 +0000 UTC m=+528.461576983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/177c030c-d783-44ae-92e9-3368ed00ef0d-tls-cert") pod "usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" (UID: "177c030c-d783-44ae-92e9-3368ed00ef0d") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 16:55:50.316536 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.316516 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/177c030c-d783-44ae-92e9-3368ed00ef0d-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b8bbs\" (UID: \"177c030c-d783-44ae-92e9-3368ed00ef0d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:50.316662 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.316645 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/177c030c-d783-44ae-92e9-3368ed00ef0d-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b8bbs\" (UID: \"177c030c-d783-44ae-92e9-3368ed00ef0d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:50.343165 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.343127 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrv58\" (UniqueName: \"kubernetes.io/projected/177c030c-d783-44ae-92e9-3368ed00ef0d-kube-api-access-wrv58\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b8bbs\" (UID: \"177c030c-d783-44ae-92e9-3368ed00ef0d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:50.638377 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.638323 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:50.821222 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.821183 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/177c030c-d783-44ae-92e9-3368ed00ef0d-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b8bbs\" (UID: \"177c030c-d783-44ae-92e9-3368ed00ef0d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:50.823760 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:50.823735 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/177c030c-d783-44ae-92e9-3368ed00ef0d-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-b8bbs\" (UID: \"177c030c-d783-44ae-92e9-3368ed00ef0d\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:51.058407 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:51.058374 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:55:51.360680 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:51.360647 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs"] Oct 30 16:55:51.361405 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:55:51.361378 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177c030c_d783_44ae_92e9_3368ed00ef0d.slice/crio-6bd9d804cd8bae517b2012437a75025c562b7ffa92bc4d4f760327e312058654 WatchSource:0}: Error finding container 6bd9d804cd8bae517b2012437a75025c562b7ffa92bc4d4f760327e312058654: Status 404 returned error can't find the container with id 6bd9d804cd8bae517b2012437a75025c562b7ffa92bc4d4f760327e312058654 Oct 30 16:55:52.032249 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:52.032208 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" event={"ID":"177c030c-d783-44ae-92e9-3368ed00ef0d","Type":"ContainerStarted","Data":"6bd9d804cd8bae517b2012437a75025c562b7ffa92bc4d4f760327e312058654"} Oct 30 16:55:53.359385 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:53.359222 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:55:58.012569 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.012532 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:58.013111 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.012623 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 16:55:58.153269 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.153077 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-qg4fp"] Oct 30 16:55:58.153467 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.153408 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-qg4fp" podUID="e043f5d8-e5d2-4f87-9797-f36165152350" containerName="manager" containerID="cri-o://f49cff2ac15f4857fad0f9bd93c669a50caf0fde1ed0639bdbcbda3b20bf0a17" gracePeriod=30 Oct 30 16:55:58.623953 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.623924 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:58.695235 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.695145 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-metrics-certs\") pod \"e043f5d8-e5d2-4f87-9797-f36165152350\" (UID: \"e043f5d8-e5d2-4f87-9797-f36165152350\") " Oct 30 16:55:58.695393 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.695261 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e043f5d8-e5d2-4f87-9797-f36165152350-kube-api-access-lmbvv\" (UniqueName: \"kubernetes.io/projected/e043f5d8-e5d2-4f87-9797-f36165152350-kube-api-access-lmbvv\") pod \"e043f5d8-e5d2-4f87-9797-f36165152350\" (UID: \"e043f5d8-e5d2-4f87-9797-f36165152350\") " Oct 30 16:55:58.695393 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.695297 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-webhook-cert\") pod \"e043f5d8-e5d2-4f87-9797-f36165152350\" (UID: \"e043f5d8-e5d2-4f87-9797-f36165152350\") " Oct 30 16:55:58.697871 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.697834 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e043f5d8-e5d2-4f87-9797-f36165152350-kube-api-access-lmbvv" pod "e043f5d8-e5d2-4f87-9797-f36165152350" (UID: "e043f5d8-e5d2-4f87-9797-f36165152350"). InnerVolumeSpecName "kube-api-access-lmbvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:58.698025 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.697944 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-metrics-certs" pod "e043f5d8-e5d2-4f87-9797-f36165152350" (UID: "e043f5d8-e5d2-4f87-9797-f36165152350"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:55:58.698205 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.698183 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-webhook-cert" pod "e043f5d8-e5d2-4f87-9797-f36165152350" (UID: "e043f5d8-e5d2-4f87-9797-f36165152350"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:55:58.797003 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.796966 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmbvv\" (UniqueName: \"kubernetes.io/projected/e043f5d8-e5d2-4f87-9797-f36165152350-kube-api-access-lmbvv\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:55:58.797003 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.796995 4828 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-webhook-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:55:58.797003 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:58.797004 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e043f5d8-e5d2-4f87-9797-f36165152350-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:55:59.078026 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:59.077985 4828 generic.go:356] "Generic (PLEG): container finished" podID="e043f5d8-e5d2-4f87-9797-f36165152350" containerID="f49cff2ac15f4857fad0f9bd93c669a50caf0fde1ed0639bdbcbda3b20bf0a17" exitCode=0 Oct 30 16:55:59.078502 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:59.078045 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-qg4fp" Oct 30 16:55:59.078502 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:59.078075 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-qg4fp" event={"ID":"e043f5d8-e5d2-4f87-9797-f36165152350","Type":"ContainerDied","Data":"f49cff2ac15f4857fad0f9bd93c669a50caf0fde1ed0639bdbcbda3b20bf0a17"} Oct 30 16:55:59.078502 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:59.078138 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-qg4fp" event={"ID":"e043f5d8-e5d2-4f87-9797-f36165152350","Type":"ContainerDied","Data":"46b9d308704dae1038e679f1398e7f4159d97e65f11bd008badaf65714b9b67c"} Oct 30 16:55:59.078502 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:59.078162 4828 scope.go:122] "RemoveContainer" containerID="f49cff2ac15f4857fad0f9bd93c669a50caf0fde1ed0639bdbcbda3b20bf0a17" Oct 30 16:55:59.092231 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:59.092210 4828 scope.go:122] "RemoveContainer" containerID="f49cff2ac15f4857fad0f9bd93c669a50caf0fde1ed0639bdbcbda3b20bf0a17" Oct 30 16:55:59.092591 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:55:59.092565 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49cff2ac15f4857fad0f9bd93c669a50caf0fde1ed0639bdbcbda3b20bf0a17\": container with ID starting with f49cff2ac15f4857fad0f9bd93c669a50caf0fde1ed0639bdbcbda3b20bf0a17 not found: ID does not exist" containerID="f49cff2ac15f4857fad0f9bd93c669a50caf0fde1ed0639bdbcbda3b20bf0a17" Oct 30 16:55:59.092653 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:59.092605 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49cff2ac15f4857fad0f9bd93c669a50caf0fde1ed0639bdbcbda3b20bf0a17"} err="failed to get container status \"f49cff2ac15f4857fad0f9bd93c669a50caf0fde1ed0639bdbcbda3b20bf0a17\": rpc error: code = NotFound desc = could not find container \"f49cff2ac15f4857fad0f9bd93c669a50caf0fde1ed0639bdbcbda3b20bf0a17\": container with ID starting with f49cff2ac15f4857fad0f9bd93c669a50caf0fde1ed0639bdbcbda3b20bf0a17 not found: ID does not exist" Oct 30 16:55:59.134878 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:59.134847 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-qg4fp"] Oct 30 16:55:59.149199 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:55:59.149164 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-qg4fp"] Oct 30 16:56:00.489538 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:00.489503 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e043f5d8-e5d2-4f87-9797-f36165152350" path="/var/lib/kubelet/pods/e043f5d8-e5d2-4f87-9797-f36165152350/volumes" Oct 30 16:56:02.096290 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:02.096248 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" event={"ID":"177c030c-d783-44ae-92e9-3368ed00ef0d","Type":"ContainerStarted","Data":"9456a4204cc95361652a0d14e87d339c5c781f28534f05c930261d17ee92aacd"} Oct 30 16:56:02.096680 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:02.096376 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:56:02.148110 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:02.148044 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" podStartSLOduration=1.993543903 podStartE2EDuration="12.148031086s" totalImagesPullingTime="10.154487183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:56:02.14709743 +0000 UTC m=+539.792607613" watchObservedRunningTime="2026-10-30 16:56:02.148031086 +0000 UTC m=+539.793541310" Oct 30 16:56:07.860537 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:07.860476 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:07.909759 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:56:07.909720 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:56:08.038639 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:08.038603 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:13.106305 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:13.106270 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" Oct 30 16:56:16.577618 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:16.577568 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:16.733082 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:16.733041 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:18.149668 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.149621 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:18.315293 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.315256 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:18.323822 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.323791 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp"] Oct 30 16:56:18.324186 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.324172 4828 cpu_manager.go:414] "containerMap: removing container" podUID="e043f5d8-e5d2-4f87-9797-f36165152350" containerName="manager" Oct 30 16:56:18.324241 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.324188 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e043f5d8-e5d2-4f87-9797-f36165152350" containerName="manager" Oct 30 16:56:18.324276 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.324262 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="e043f5d8-e5d2-4f87-9797-f36165152350" containerName="manager" Oct 30 16:56:18.327764 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.327745 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.345721 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.345688 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cdj9d\"" Oct 30 16:56:18.358978 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.358944 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 16:56:18.358978 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.358944 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 16:56:18.363564 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.363539 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp"] Oct 30 16:56:18.466997 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.466890 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-metrics-certs\") pod \"maas-api-5c48877c75-vcmmp\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.467169 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.467004 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbknx\" (UniqueName: \"kubernetes.io/projected/3563a565-f8f5-44b3-b969-a14c15a09155-kube-api-access-hbknx\") pod \"maas-api-5c48877c75-vcmmp\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.467169 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.467041 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3563a565-f8f5-44b3-b969-a14c15a09155-openshift-service-ca\") pod \"maas-api-5c48877c75-vcmmp\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.467169 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.467068 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-maas-api-tls\") pod \"maas-api-5c48877c75-vcmmp\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.470165 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.470138 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:56:18.567953 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.567873 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-metrics-certs\") pod \"maas-api-5c48877c75-vcmmp\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.568157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.567968 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbknx\" (UniqueName: \"kubernetes.io/projected/3563a565-f8f5-44b3-b969-a14c15a09155-kube-api-access-hbknx\") pod \"maas-api-5c48877c75-vcmmp\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.568157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.568006 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3563a565-f8f5-44b3-b969-a14c15a09155-openshift-service-ca\") pod \"maas-api-5c48877c75-vcmmp\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.568157 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.568033 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-maas-api-tls\") pod \"maas-api-5c48877c75-vcmmp\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.568723 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.568703 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3563a565-f8f5-44b3-b969-a14c15a09155-openshift-service-ca\") pod \"maas-api-5c48877c75-vcmmp\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.570530 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.570507 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-maas-api-tls\") pod \"maas-api-5c48877c75-vcmmp\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.570630 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.570549 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-metrics-certs\") pod \"maas-api-5c48877c75-vcmmp\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.592861 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.592828 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbknx\" (UniqueName: \"kubernetes.io/projected/3563a565-f8f5-44b3-b969-a14c15a09155-kube-api-access-hbknx\") pod \"maas-api-5c48877c75-vcmmp\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.641005 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.640964 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:18.954029 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:18.954000 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp"] Oct 30 16:56:18.954914 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:56:18.954872 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3563a565_f8f5_44b3_b969_a14c15a09155.slice/crio-7865bfcff7681d0f6fec95d75cd5b5a949b49b5e75f5838afe6121ccf9d3ba0c WatchSource:0}: Error finding container 7865bfcff7681d0f6fec95d75cd5b5a949b49b5e75f5838afe6121ccf9d3ba0c: Status 404 returned error can't find the container with id 7865bfcff7681d0f6fec95d75cd5b5a949b49b5e75f5838afe6121ccf9d3ba0c Oct 30 16:56:19.180870 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:19.180812 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" event={"ID":"3563a565-f8f5-44b3-b969-a14c15a09155","Type":"ContainerStarted","Data":"7865bfcff7681d0f6fec95d75cd5b5a949b49b5e75f5838afe6121ccf9d3ba0c"} Oct 30 16:56:21.201513 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:21.201471 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" event={"ID":"3563a565-f8f5-44b3-b969-a14c15a09155","Type":"ContainerStarted","Data":"c7bf51fe92fbe0f9088934ec0cd8ca28f9143265469b6b2c89adf4b0e2ae0833"} Oct 30 16:56:21.201970 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:21.201602 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:21.248541 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:21.248487 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" podStartSLOduration=1.830716844 podStartE2EDuration="3.248471347s" totalImagesPullingTime="1.417754503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:56:21.246332358 +0000 UTC m=+558.891842537" watchObservedRunningTime="2026-10-30 16:56:21.248471347 +0000 UTC m=+558.893981527" Oct 30 16:56:27.212741 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:27.212708 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 16:56:30.251441 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:30.251407 4828 generic.go:356] "Generic (PLEG): container finished" podID="1fc9a9e7-416b-4b87-912e-a524f41e0d53" containerID="5ae5a187df95bc23f1d80be94a3b53ae9f4a9c96ac5d71102467d639a3049f42" exitCode=0 Oct 30 16:56:30.251880 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:30.251484 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" event={"ID":"1fc9a9e7-416b-4b87-912e-a524f41e0d53","Type":"ContainerDied","Data":"5ae5a187df95bc23f1d80be94a3b53ae9f4a9c96ac5d71102467d639a3049f42"} Oct 30 16:56:30.251880 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:30.251864 4828 scope.go:122] "RemoveContainer" containerID="5ae5a187df95bc23f1d80be94a3b53ae9f4a9c96ac5d71102467d639a3049f42" Oct 30 16:56:30.252887 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:30.252861 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:56:30.252981 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:30.252947 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:56:30.253018 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:30.252979 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:56:31.258093 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:31.258059 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" event={"ID":"1fc9a9e7-416b-4b87-912e-a524f41e0d53","Type":"ContainerStarted","Data":"507e991d491c0127632b21b299194772aa5a32bc6d14e17d441e289c0a4dac7f"} Oct 30 16:56:31.331641 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:31.331600 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:56:31.331861 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:31.331818 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r26lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:56:31.331942 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:31.331857 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" podUID="1fc9a9e7-416b-4b87-912e-a524f41e0d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 16:56:32.332203 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:32.332165 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r26lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:56:32.332666 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:32.332222 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" podUID="1fc9a9e7-416b-4b87-912e-a524f41e0d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 16:56:33.332091 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:33.332057 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r26lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:56:33.332261 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:33.332106 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" podUID="1fc9a9e7-416b-4b87-912e-a524f41e0d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 16:56:34.332670 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:34.332632 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r26lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:56:34.333092 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:34.332693 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" podUID="1fc9a9e7-416b-4b87-912e-a524f41e0d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 16:56:35.332432 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:35.332395 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r26lb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:56:35.332635 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:35.332454 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" podUID="1fc9a9e7-416b-4b87-912e-a524f41e0d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 16:56:36.335506 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:36.335474 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:56:36.335871 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:36.335753 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:56:36.336367 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:36.336349 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" Oct 30 16:56:37.909335 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:56:37.909297 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:56:53.491468 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:56:53.491429 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:02.078910 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:02.078866 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:02.228151 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:02.228119 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:02.537513 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:02.537486 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 16:57:02.538913 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:02.538876 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 16:57:07.910160 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:57:07.910123 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:57:20.364246 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.364212 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r"] Oct 30 16:57:20.367780 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.367758 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.387147 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.387121 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 16:57:20.396886 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.396864 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 16:57:20.397018 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.396864 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-45tpm\"" Oct 30 16:57:20.397228 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.397211 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 16:57:20.402570 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.402551 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r"] Oct 30 16:57:20.476731 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.476691 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:20.516270 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.516234 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j6rl\" (UniqueName: \"kubernetes.io/projected/6ab66bcb-c480-4c94-abe5-307e17932f99-kube-api-access-7j6rl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.516512 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.516280 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab66bcb-c480-4c94-abe5-307e17932f99-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.516512 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.516301 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab66bcb-c480-4c94-abe5-307e17932f99-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.516512 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.516431 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab66bcb-c480-4c94-abe5-307e17932f99-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.516512 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.516484 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab66bcb-c480-4c94-abe5-307e17932f99-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.516666 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.516562 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab66bcb-c480-4c94-abe5-307e17932f99-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.617290 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.617212 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j6rl\" (UniqueName: \"kubernetes.io/projected/6ab66bcb-c480-4c94-abe5-307e17932f99-kube-api-access-7j6rl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.617290 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.617259 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab66bcb-c480-4c94-abe5-307e17932f99-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.617290 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.617278 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab66bcb-c480-4c94-abe5-307e17932f99-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.617551 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.617295 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab66bcb-c480-4c94-abe5-307e17932f99-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.617551 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.617397 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab66bcb-c480-4c94-abe5-307e17932f99-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.617551 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.617457 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab66bcb-c480-4c94-abe5-307e17932f99-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.617724 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.617704 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab66bcb-c480-4c94-abe5-307e17932f99-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.617820 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.617791 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab66bcb-c480-4c94-abe5-307e17932f99-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.617967 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.617836 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab66bcb-c480-4c94-abe5-307e17932f99-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.619656 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.619632 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab66bcb-c480-4c94-abe5-307e17932f99-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.619840 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.619827 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab66bcb-c480-4c94-abe5-307e17932f99-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.641527 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.641496 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j6rl\" (UniqueName: \"kubernetes.io/projected/6ab66bcb-c480-4c94-abe5-307e17932f99-kube-api-access-7j6rl\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r\" (UID: \"6ab66bcb-c480-4c94-abe5-307e17932f99\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.681143 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.681110 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:20.993221 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.993192 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r"] Oct 30 16:57:20.994528 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:57:20.994501 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab66bcb_c480_4c94_abe5_307e17932f99.slice/crio-af323c036061c848973d7616c915242a5f7dbafaee5dbdc4fe7892758a431633 WatchSource:0}: Error finding container af323c036061c848973d7616c915242a5f7dbafaee5dbdc4fe7892758a431633: Status 404 returned error can't find the container with id af323c036061c848973d7616c915242a5f7dbafaee5dbdc4fe7892758a431633 Oct 30 16:57:20.995940 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:20.995890 4828 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:57:21.546052 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:21.545958 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" event={"ID":"6ab66bcb-c480-4c94-abe5-307e17932f99","Type":"ContainerStarted","Data":"af323c036061c848973d7616c915242a5f7dbafaee5dbdc4fe7892758a431633"} Oct 30 16:57:22.552261 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:22.552222 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" event={"ID":"6ab66bcb-c480-4c94-abe5-307e17932f99","Type":"ContainerStarted","Data":"19b6dba02c898c03b3c3616267326a7acf7c1a5e5194da5366b89fcc18464265"} Oct 30 16:57:22.598942 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:22.598879 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" podStartSLOduration=1.411533472 podStartE2EDuration="2.598862779s" totalImagesPullingTime="1.187329307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:22.597147369 +0000 UTC m=+620.242657549" watchObservedRunningTime="2026-10-30 16:57:22.598862779 +0000 UTC m=+620.244372959" Oct 30 16:57:22.682207 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:22.682174 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:23.262863 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:23.262829 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:25.450919 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.450870 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8"] Oct 30 16:57:25.455187 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.455159 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.462539 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.462515 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86d11598-b8b5-4653-93ab-438cf26e63eb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.462686 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.462546 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86d11598-b8b5-4653-93ab-438cf26e63eb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.462686 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.462581 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d11598-b8b5-4653-93ab-438cf26e63eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.462810 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.462758 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d11598-b8b5-4653-93ab-438cf26e63eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.462874 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.462798 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62q6z\" (UniqueName: \"kubernetes.io/projected/86d11598-b8b5-4653-93ab-438cf26e63eb-kube-api-access-62q6z\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.462970 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.462870 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86d11598-b8b5-4653-93ab-438cf26e63eb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.485938 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.485887 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 16:57:25.488056 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.488031 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8"] Oct 30 16:57:25.548622 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.548574 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:25.563662 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.563635 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d11598-b8b5-4653-93ab-438cf26e63eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.563799 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.563694 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62q6z\" (UniqueName: \"kubernetes.io/projected/86d11598-b8b5-4653-93ab-438cf26e63eb-kube-api-access-62q6z\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.563863 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.563793 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86d11598-b8b5-4653-93ab-438cf26e63eb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.563863 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.563834 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86d11598-b8b5-4653-93ab-438cf26e63eb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.564137 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.564112 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86d11598-b8b5-4653-93ab-438cf26e63eb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.564212 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.564167 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d11598-b8b5-4653-93ab-438cf26e63eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.564449 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.564415 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86d11598-b8b5-4653-93ab-438cf26e63eb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.564557 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.564480 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86d11598-b8b5-4653-93ab-438cf26e63eb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.564557 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.564492 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86d11598-b8b5-4653-93ab-438cf26e63eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.566185 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.566161 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86d11598-b8b5-4653-93ab-438cf26e63eb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.566577 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.566554 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86d11598-b8b5-4653-93ab-438cf26e63eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.592050 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.592027 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62q6z\" (UniqueName: \"kubernetes.io/projected/86d11598-b8b5-4653-93ab-438cf26e63eb-kube-api-access-62q6z\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-p7tp8\" (UID: \"86d11598-b8b5-4653-93ab-438cf26e63eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:25.771145 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:25.771104 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:26.071954 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:26.071925 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8"] Oct 30 16:57:26.072432 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:57:26.072408 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d11598_b8b5_4653_93ab_438cf26e63eb.slice/crio-23792f0ba875f20a879bcb8d6a335bb5404168a3ca8323962b17ac46d38cdc5a WatchSource:0}: Error finding container 23792f0ba875f20a879bcb8d6a335bb5404168a3ca8323962b17ac46d38cdc5a: Status 404 returned error can't find the container with id 23792f0ba875f20a879bcb8d6a335bb5404168a3ca8323962b17ac46d38cdc5a Oct 30 16:57:26.575742 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:26.575711 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" event={"ID":"86d11598-b8b5-4653-93ab-438cf26e63eb","Type":"ContainerStarted","Data":"469bd69604cac5297a11387042abf9df78c45a120269c9b70db5e27ed4c76276"} Oct 30 16:57:26.576141 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:26.575761 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" event={"ID":"86d11598-b8b5-4653-93ab-438cf26e63eb","Type":"ContainerStarted","Data":"23792f0ba875f20a879bcb8d6a335bb5404168a3ca8323962b17ac46d38cdc5a"} Oct 30 16:57:26.623532 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:26.623427 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" podStartSLOduration=1.3968809119999999 podStartE2EDuration="1.623413808s" totalImagesPullingTime="226.532896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:26.622811994 +0000 UTC m=+624.268322176" watchObservedRunningTime="2026-10-30 16:57:26.623413808 +0000 UTC m=+624.268923988" Oct 30 16:57:26.771366 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:26.771329 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:27.985758 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:27.985726 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:30.682088 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:30.682047 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:30.698149 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:30.698114 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:31.201697 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.200721 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp"] Oct 30 16:57:31.206393 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.206367 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.211338 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.211311 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/204a77c5-f77b-4842-ae59-cfe3c86cd976-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.211473 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.211352 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/204a77c5-f77b-4842-ae59-cfe3c86cd976-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.211473 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.211427 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/204a77c5-f77b-4842-ae59-cfe3c86cd976-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.211473 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.211450 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/204a77c5-f77b-4842-ae59-cfe3c86cd976-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.211473 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.211468 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/204a77c5-f77b-4842-ae59-cfe3c86cd976-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.211699 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.211601 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qshg4\" (UniqueName: \"kubernetes.io/projected/204a77c5-f77b-4842-ae59-cfe3c86cd976-kube-api-access-qshg4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.222314 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.222288 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 16:57:31.238575 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.238547 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp"] Oct 30 16:57:31.298230 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.298189 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:31.312491 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.312462 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/204a77c5-f77b-4842-ae59-cfe3c86cd976-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.312665 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.312503 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/204a77c5-f77b-4842-ae59-cfe3c86cd976-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.312665 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.312532 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/204a77c5-f77b-4842-ae59-cfe3c86cd976-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.312665 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.312602 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qshg4\" (UniqueName: \"kubernetes.io/projected/204a77c5-f77b-4842-ae59-cfe3c86cd976-kube-api-access-qshg4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.312833 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.312667 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/204a77c5-f77b-4842-ae59-cfe3c86cd976-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.312833 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.312711 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/204a77c5-f77b-4842-ae59-cfe3c86cd976-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.312956 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.312936 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/204a77c5-f77b-4842-ae59-cfe3c86cd976-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.313052 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.313026 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/204a77c5-f77b-4842-ae59-cfe3c86cd976-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.313112 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.313045 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/204a77c5-f77b-4842-ae59-cfe3c86cd976-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.314790 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.314760 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/204a77c5-f77b-4842-ae59-cfe3c86cd976-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.315326 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.315304 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/204a77c5-f77b-4842-ae59-cfe3c86cd976-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.337611 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.337570 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qshg4\" (UniqueName: \"kubernetes.io/projected/204a77c5-f77b-4842-ae59-cfe3c86cd976-kube-api-access-qshg4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp\" (UID: \"204a77c5-f77b-4842-ae59-cfe3c86cd976\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.526068 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.526027 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:31.622802 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.622774 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" Oct 30 16:57:31.839519 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:31.839486 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp"] Oct 30 16:57:31.840254 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:57:31.840227 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204a77c5_f77b_4842_ae59_cfe3c86cd976.slice/crio-2e54814b4ee44f52ab1d2bd06d36b01e4e7bff4ff096aa8b25e7a65927fbd3c9 WatchSource:0}: Error finding container 2e54814b4ee44f52ab1d2bd06d36b01e4e7bff4ff096aa8b25e7a65927fbd3c9: Status 404 returned error can't find the container with id 2e54814b4ee44f52ab1d2bd06d36b01e4e7bff4ff096aa8b25e7a65927fbd3c9 Oct 30 16:57:32.618008 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:32.617970 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" event={"ID":"204a77c5-f77b-4842-ae59-cfe3c86cd976","Type":"ContainerStarted","Data":"8a1f4fe36b0217d7357c91e270924af029aeaf9874c8510b8345b48f0e89ff36"} Oct 30 16:57:32.618220 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:32.618033 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" event={"ID":"204a77c5-f77b-4842-ae59-cfe3c86cd976","Type":"ContainerStarted","Data":"2e54814b4ee44f52ab1d2bd06d36b01e4e7bff4ff096aa8b25e7a65927fbd3c9"} Oct 30 16:57:32.665488 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:32.665425 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" podStartSLOduration=1.465662312 podStartE2EDuration="1.665411012s" totalImagesPullingTime="199.7487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:32.664199026 +0000 UTC m=+630.309709207" watchObservedRunningTime="2026-10-30 16:57:32.665411012 +0000 UTC m=+630.310921192" Oct 30 16:57:33.385607 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:33.385538 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:33.526538 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:33.526488 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:35.385360 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:35.385316 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:35.771660 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:35.771620 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:35.786225 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:35.786186 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:36.654106 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:36.654078 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" Oct 30 16:57:37.910196 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:57:37.910152 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:57:41.526768 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:41.526710 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:41.539692 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:41.539661 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:41.686225 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:41.686195 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" Oct 30 16:57:48.133944 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.133884 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l"] Oct 30 16:57:48.137379 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.137362 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.171859 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.171825 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l"] Oct 30 16:57:48.270055 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.270020 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.270055 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.270055 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.270262 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.270169 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.270262 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.270216 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thdfb\" (UniqueName: \"kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-kube-api-access-thdfb\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.270331 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.270281 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.270331 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.270300 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab34194-be4a-48ba-9b6d-c759be67f22d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.270395 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.270350 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.270395 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.270371 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.270456 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.270400 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.371549 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.371513 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.371549 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.371548 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.371785 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.371606 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.371785 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.371624 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thdfb\" (UniqueName: \"kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-kube-api-access-thdfb\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.371785 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.371643 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.371785 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.371664 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab34194-be4a-48ba-9b6d-c759be67f22d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.371785 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.371690 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.371785 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.371722 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.371785 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.371752 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.372150 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.372083 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.372240 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.372219 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.372300 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.372247 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.372300 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.372280 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.372632 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.372604 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab34194-be4a-48ba-9b6d-c759be67f22d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.373878 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.373859 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.374357 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.374338 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.396808 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.396741 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.397094 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.397073 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thdfb\" (UniqueName: \"kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-kube-api-access-thdfb\") pod \"maas-default-gateway-openshift-default-6db586dd6c-5x69l\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.452110 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.452076 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:48.767264 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.767236 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l"] Oct 30 16:57:48.768975 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:57:48.768950 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab34194_be4a_48ba_9b6d_c759be67f22d.slice/crio-efb23865fb8ed8c5658f4db5a0125f8b0df3504cb5ce8f68087ebd4820d107d1 WatchSource:0}: Error finding container efb23865fb8ed8c5658f4db5a0125f8b0df3504cb5ce8f68087ebd4820d107d1: Status 404 returned error can't find the container with id efb23865fb8ed8c5658f4db5a0125f8b0df3504cb5ce8f68087ebd4820d107d1 Oct 30 16:57:48.771207 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.771178 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:57:48.771291 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.771253 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:57:48.771291 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:48.771280 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:57:49.719351 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:49.719308 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" event={"ID":"0ab34194-be4a-48ba-9b6d-c759be67f22d","Type":"ContainerStarted","Data":"c158340e8474f363e366ee96ef8532afa83a49e0f1a10dad610f51a9748c1167"} Oct 30 16:57:49.719755 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:49.719371 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" event={"ID":"0ab34194-be4a-48ba-9b6d-c759be67f22d","Type":"ContainerStarted","Data":"efb23865fb8ed8c5658f4db5a0125f8b0df3504cb5ce8f68087ebd4820d107d1"} Oct 30 16:57:49.792927 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:49.782889 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" podStartSLOduration=1.782871418 podStartE2EDuration="1.782871418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:49.781561831 +0000 UTC m=+647.427072011" watchObservedRunningTime="2026-10-30 16:57:49.782871418 +0000 UTC m=+647.428381600" Oct 30 16:57:50.453145 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:50.453103 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:50.454887 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:50.454863 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5x69l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:57:50.455031 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:50.454926 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" podUID="0ab34194-be4a-48ba-9b6d-c759be67f22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:57:51.452811 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:51.452773 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5x69l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:57:51.453256 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:51.452831 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" podUID="0ab34194-be4a-48ba-9b6d-c759be67f22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:57:52.452753 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:52.452712 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5x69l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:57:52.452998 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:52.452772 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" podUID="0ab34194-be4a-48ba-9b6d-c759be67f22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:57:53.398203 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.398152 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-lsdd7"] Oct 30 16:57:53.402534 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.402459 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" Oct 30 16:57:53.418958 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.418932 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-lsdd7"] Oct 30 16:57:53.427965 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.426857 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 16:57:53.427965 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.427818 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-lfcst\"" Oct 30 16:57:53.436732 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.436353 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:53.438258 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.438238 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gxqq\" (UniqueName: \"kubernetes.io/projected/006b690d-277e-412c-b8c5-962660f560e3-kube-api-access-5gxqq\") pod \"ai-gateway-controller-66dd76bf94-lsdd7\" (UID: \"006b690d-277e-412c-b8c5-962660f560e3\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" Oct 30 16:57:53.438480 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.438466 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006b690d-277e-412c-b8c5-962660f560e3-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-lsdd7\" (UID: \"006b690d-277e-412c-b8c5-962660f560e3\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" Oct 30 16:57:53.454473 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.454096 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5x69l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:57:53.454473 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.454146 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" podUID="0ab34194-be4a-48ba-9b6d-c759be67f22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:57:53.501932 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.501291 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 16:57:53.539629 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.539593 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gxqq\" (UniqueName: \"kubernetes.io/projected/006b690d-277e-412c-b8c5-962660f560e3-kube-api-access-5gxqq\") pod \"ai-gateway-controller-66dd76bf94-lsdd7\" (UID: \"006b690d-277e-412c-b8c5-962660f560e3\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" Oct 30 16:57:53.539798 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.539660 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006b690d-277e-412c-b8c5-962660f560e3-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-lsdd7\" (UID: \"006b690d-277e-412c-b8c5-962660f560e3\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" Oct 30 16:57:53.542260 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.542237 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006b690d-277e-412c-b8c5-962660f560e3-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-lsdd7\" (UID: \"006b690d-277e-412c-b8c5-962660f560e3\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" Oct 30 16:57:53.595711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.595641 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gxqq\" (UniqueName: \"kubernetes.io/projected/006b690d-277e-412c-b8c5-962660f560e3-kube-api-access-5gxqq\") pod \"ai-gateway-controller-66dd76bf94-lsdd7\" (UID: \"006b690d-277e-412c-b8c5-962660f560e3\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" Oct 30 16:57:53.609196 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.609164 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:53.719122 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.719036 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" Oct 30 16:57:53.785187 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:53.780772 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:54.049843 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:54.049814 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-lsdd7"] Oct 30 16:57:54.052885 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:57:54.052844 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006b690d_277e_412c_b8c5_962660f560e3.slice/crio-9c9b93b82aa28a66054df3f63f3f6822a240c0232cffef57cc3953277ea1bb47 WatchSource:0}: Error finding container 9c9b93b82aa28a66054df3f63f3f6822a240c0232cffef57cc3953277ea1bb47: Status 404 returned error can't find the container with id 9c9b93b82aa28a66054df3f63f3f6822a240c0232cffef57cc3953277ea1bb47 Oct 30 16:57:54.452809 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:54.452727 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5x69l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:57:54.452809 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:54.452783 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" podUID="0ab34194-be4a-48ba-9b6d-c759be67f22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:57:54.752207 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:54.752050 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" event={"ID":"006b690d-277e-412c-b8c5-962660f560e3","Type":"ContainerStarted","Data":"9c9b93b82aa28a66054df3f63f3f6822a240c0232cffef57cc3953277ea1bb47"} Oct 30 16:57:55.452894 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:55.452853 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5x69l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:57:55.453114 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:55.452933 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" podUID="0ab34194-be4a-48ba-9b6d-c759be67f22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:57:56.453417 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:56.453375 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-5x69l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:57:56.453934 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:56.453441 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" podUID="0ab34194-be4a-48ba-9b6d-c759be67f22d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:57:57.551706 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:57.551660 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:57.552361 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:57.552314 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:57.553259 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:57.553239 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 16:57:57.772627 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:57.772589 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" event={"ID":"006b690d-277e-412c-b8c5-962660f560e3","Type":"ContainerStarted","Data":"9db1bfb28e6fcf31507297eac36c1ab485694607768c188bffd1c0f7a5ce2284"} Oct 30 16:57:57.818323 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:57.818211 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" podStartSLOduration=2.143868459 podStartE2EDuration="4.818196226s" totalImagesPullingTime="2.674327767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:57.816203738 +0000 UTC m=+655.461713917" watchObservedRunningTime="2026-10-30 16:57:57.818196226 +0000 UTC m=+655.463706436" Oct 30 16:57:57.996395 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:57.996352 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:58.065590 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.065545 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-mj8j7"] Oct 30 16:57:58.069420 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.069362 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-mj8j7" Oct 30 16:57:58.090926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.090725 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 16:57:58.091252 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.090946 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-qfpwd\"" Oct 30 16:57:58.091252 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.091231 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-mj8j7"] Oct 30 16:57:58.128788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.128757 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk"] Oct 30 16:57:58.132482 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.132456 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk" Oct 30 16:57:58.164947 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.162469 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk"] Oct 30 16:57:58.185095 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.183676 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhfgk\" (UniqueName: \"kubernetes.io/projected/a7513be4-9787-41d1-99a1-29ffc37bc75c-kube-api-access-nhfgk\") pod \"payload-pre-processing-7b98bb678c-w9xlk\" (UID: \"a7513be4-9787-41d1-99a1-29ffc37bc75c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk" Oct 30 16:57:58.185095 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.183736 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7513be4-9787-41d1-99a1-29ffc37bc75c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-w9xlk\" (UID: \"a7513be4-9787-41d1-99a1-29ffc37bc75c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk" Oct 30 16:57:58.185095 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.183836 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skgbh\" (UniqueName: \"kubernetes.io/projected/c055e279-12c7-4707-b602-56548f5df870-kube-api-access-skgbh\") pod \"payload-processing-5f78687d5b-mj8j7\" (UID: \"c055e279-12c7-4707-b602-56548f5df870\") " pod="openshift-ingress/payload-processing-5f78687d5b-mj8j7" Oct 30 16:57:58.185095 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.183872 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c055e279-12c7-4707-b602-56548f5df870-plugins-config-volume\") pod \"payload-processing-5f78687d5b-mj8j7\" (UID: \"c055e279-12c7-4707-b602-56548f5df870\") " pod="openshift-ingress/payload-processing-5f78687d5b-mj8j7" Oct 30 16:57:58.185095 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.184312 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:57:58.285100 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.285051 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfgk\" (UniqueName: \"kubernetes.io/projected/a7513be4-9787-41d1-99a1-29ffc37bc75c-kube-api-access-nhfgk\") pod \"payload-pre-processing-7b98bb678c-w9xlk\" (UID: \"a7513be4-9787-41d1-99a1-29ffc37bc75c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk" Oct 30 16:57:58.285305 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.285141 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7513be4-9787-41d1-99a1-29ffc37bc75c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-w9xlk\" (UID: \"a7513be4-9787-41d1-99a1-29ffc37bc75c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk" Oct 30 16:57:58.285305 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.285267 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skgbh\" (UniqueName: \"kubernetes.io/projected/c055e279-12c7-4707-b602-56548f5df870-kube-api-access-skgbh\") pod \"payload-processing-5f78687d5b-mj8j7\" (UID: \"c055e279-12c7-4707-b602-56548f5df870\") " pod="openshift-ingress/payload-processing-5f78687d5b-mj8j7" Oct 30 16:57:58.285418 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.285309 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c055e279-12c7-4707-b602-56548f5df870-plugins-config-volume\") pod \"payload-processing-5f78687d5b-mj8j7\" (UID: \"c055e279-12c7-4707-b602-56548f5df870\") " pod="openshift-ingress/payload-processing-5f78687d5b-mj8j7" Oct 30 16:57:58.286139 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.286114 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7513be4-9787-41d1-99a1-29ffc37bc75c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-w9xlk\" (UID: \"a7513be4-9787-41d1-99a1-29ffc37bc75c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk" Oct 30 16:57:58.286329 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.286305 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c055e279-12c7-4707-b602-56548f5df870-plugins-config-volume\") pod \"payload-processing-5f78687d5b-mj8j7\" (UID: \"c055e279-12c7-4707-b602-56548f5df870\") " pod="openshift-ingress/payload-processing-5f78687d5b-mj8j7" Oct 30 16:57:58.310160 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.310127 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skgbh\" (UniqueName: \"kubernetes.io/projected/c055e279-12c7-4707-b602-56548f5df870-kube-api-access-skgbh\") pod \"payload-processing-5f78687d5b-mj8j7\" (UID: \"c055e279-12c7-4707-b602-56548f5df870\") " pod="openshift-ingress/payload-processing-5f78687d5b-mj8j7" Oct 30 16:57:58.310509 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.310484 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfgk\" (UniqueName: \"kubernetes.io/projected/a7513be4-9787-41d1-99a1-29ffc37bc75c-kube-api-access-nhfgk\") pod \"payload-pre-processing-7b98bb678c-w9xlk\" (UID: \"a7513be4-9787-41d1-99a1-29ffc37bc75c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk" Oct 30 16:57:58.383250 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.383155 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-mj8j7" Oct 30 16:57:58.445700 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.445659 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk" Oct 30 16:57:58.703254 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.701702 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-mj8j7"] Oct 30 16:57:58.703254 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:57:58.702211 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc055e279_12c7_4707_b602_56548f5df870.slice/crio-fa284aa272939f1b45975c91b3606d6446f67ebea214eec09e1a11580e631469 WatchSource:0}: Error finding container fa284aa272939f1b45975c91b3606d6446f67ebea214eec09e1a11580e631469: Status 404 returned error can't find the container with id fa284aa272939f1b45975c91b3606d6446f67ebea214eec09e1a11580e631469 Oct 30 16:57:58.783177 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.783115 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk"] Oct 30 16:57:58.784722 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.784658 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-mj8j7" event={"ID":"c055e279-12c7-4707-b602-56548f5df870","Type":"ContainerStarted","Data":"fa284aa272939f1b45975c91b3606d6446f67ebea214eec09e1a11580e631469"} Oct 30 16:57:58.785201 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:58.785146 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" Oct 30 16:57:58.786912 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:57:58.786881 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7513be4_9787_41d1_99a1_29ffc37bc75c.slice/crio-b6a8748d9cbe65495338a6332e41cf1f92f6b19abddff185c783c4ae3e0a9a28 WatchSource:0}: Error finding container b6a8748d9cbe65495338a6332e41cf1f92f6b19abddff185c783c4ae3e0a9a28: Status 404 returned error can't find the container with id b6a8748d9cbe65495338a6332e41cf1f92f6b19abddff185c783c4ae3e0a9a28 Oct 30 16:57:59.796395 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:57:59.796352 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk" event={"ID":"a7513be4-9787-41d1-99a1-29ffc37bc75c","Type":"ContainerStarted","Data":"b6a8748d9cbe65495338a6332e41cf1f92f6b19abddff185c783c4ae3e0a9a28"} Oct 30 16:58:01.809244 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:01.809204 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-mj8j7" event={"ID":"c055e279-12c7-4707-b602-56548f5df870","Type":"ContainerStarted","Data":"01bd38a27986a1f5e876b280b9a36cfbaae16c77512f48eecf0be9b513b9a68f"} Oct 30 16:58:01.809729 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:01.809268 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-mj8j7" Oct 30 16:58:01.810855 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:01.810830 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk" event={"ID":"a7513be4-9787-41d1-99a1-29ffc37bc75c","Type":"ContainerStarted","Data":"246f55bfec50303915b9e0547fe8b865f3ac1636eb7e27ae000728809fa7bcfb"} Oct 30 16:58:01.811034 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:01.810929 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk" Oct 30 16:58:01.853863 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:01.853803 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-mj8j7" podStartSLOduration=1.032408434 podStartE2EDuration="3.853786168s" totalImagesPullingTime="2.821377734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:01.85235482 +0000 UTC m=+659.497865002" watchObservedRunningTime="2026-10-30 16:58:01.853786168 +0000 UTC m=+659.499296349" Oct 30 16:58:01.896524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:01.896469 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk" podStartSLOduration=1.225258092 podStartE2EDuration="3.896454517s" totalImagesPullingTime="2.671196425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:01.895290764 +0000 UTC m=+659.540800945" watchObservedRunningTime="2026-10-30 16:58:01.896454517 +0000 UTC m=+659.541964696" Oct 30 16:58:07.910108 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:58:07.910070 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:58:09.798726 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:09.798697 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" Oct 30 16:58:10.094326 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:10.094244 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-575996c6dd-2tngr"] Oct 30 16:58:10.099402 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:10.099382 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-575996c6dd-2tngr" Oct 30 16:58:10.118838 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:10.118801 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-575996c6dd-2tngr"] Oct 30 16:58:10.214000 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:10.213964 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21c11229-9597-4027-b247-2c5ad341a24b-metrics-certs\") pod \"ai-gateway-controller-575996c6dd-2tngr\" (UID: \"21c11229-9597-4027-b247-2c5ad341a24b\") " pod="opendatahub/ai-gateway-controller-575996c6dd-2tngr" Oct 30 16:58:10.214190 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:10.214025 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmdt7\" (UniqueName: \"kubernetes.io/projected/21c11229-9597-4027-b247-2c5ad341a24b-kube-api-access-hmdt7\") pod \"ai-gateway-controller-575996c6dd-2tngr\" (UID: \"21c11229-9597-4027-b247-2c5ad341a24b\") " pod="opendatahub/ai-gateway-controller-575996c6dd-2tngr" Oct 30 16:58:10.315270 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:10.315230 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmdt7\" (UniqueName: \"kubernetes.io/projected/21c11229-9597-4027-b247-2c5ad341a24b-kube-api-access-hmdt7\") pod \"ai-gateway-controller-575996c6dd-2tngr\" (UID: \"21c11229-9597-4027-b247-2c5ad341a24b\") " pod="opendatahub/ai-gateway-controller-575996c6dd-2tngr" Oct 30 16:58:10.315438 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:10.315342 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21c11229-9597-4027-b247-2c5ad341a24b-metrics-certs\") pod \"ai-gateway-controller-575996c6dd-2tngr\" (UID: \"21c11229-9597-4027-b247-2c5ad341a24b\") " pod="opendatahub/ai-gateway-controller-575996c6dd-2tngr" Oct 30 16:58:10.317772 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:10.317752 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21c11229-9597-4027-b247-2c5ad341a24b-metrics-certs\") pod \"ai-gateway-controller-575996c6dd-2tngr\" (UID: \"21c11229-9597-4027-b247-2c5ad341a24b\") " pod="opendatahub/ai-gateway-controller-575996c6dd-2tngr" Oct 30 16:58:10.339851 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:10.339822 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmdt7\" (UniqueName: \"kubernetes.io/projected/21c11229-9597-4027-b247-2c5ad341a24b-kube-api-access-hmdt7\") pod \"ai-gateway-controller-575996c6dd-2tngr\" (UID: \"21c11229-9597-4027-b247-2c5ad341a24b\") " pod="opendatahub/ai-gateway-controller-575996c6dd-2tngr" Oct 30 16:58:10.412614 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:10.412541 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-575996c6dd-2tngr" Oct 30 16:58:10.713727 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:10.713700 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-575996c6dd-2tngr"] Oct 30 16:58:10.714740 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:58:10.714710 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c11229_9597_4027_b247_2c5ad341a24b.slice/crio-4bb2c2ec439e090117a3dc6f402cbfd3c41a195b1bad059afadb2c22dc7c95cf WatchSource:0}: Error finding container 4bb2c2ec439e090117a3dc6f402cbfd3c41a195b1bad059afadb2c22dc7c95cf: Status 404 returned error can't find the container with id 4bb2c2ec439e090117a3dc6f402cbfd3c41a195b1bad059afadb2c22dc7c95cf Oct 30 16:58:10.863165 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:10.863129 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-575996c6dd-2tngr" event={"ID":"21c11229-9597-4027-b247-2c5ad341a24b","Type":"ContainerStarted","Data":"4bb2c2ec439e090117a3dc6f402cbfd3c41a195b1bad059afadb2c22dc7c95cf"} Oct 30 16:58:11.869596 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:11.869557 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-575996c6dd-2tngr" event={"ID":"21c11229-9597-4027-b247-2c5ad341a24b","Type":"ContainerStarted","Data":"8c8571488a86123fcd09fa6f1b84e255dff26fe7422b85bf87dd3144897a2b86"} Oct 30 16:58:11.870109 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:11.869681 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-575996c6dd-2tngr" Oct 30 16:58:11.915392 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:11.915335 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-575996c6dd-2tngr" podStartSLOduration=1.27593872 podStartE2EDuration="1.915319795s" totalImagesPullingTime="639.381075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:11.913207619 +0000 UTC m=+669.558717804" watchObservedRunningTime="2026-10-30 16:58:11.915319795 +0000 UTC m=+669.560829999" Oct 30 16:58:12.820132 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:12.820103 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w9xlk" Oct 30 16:58:12.820429 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:12.820216 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-mj8j7" Oct 30 16:58:22.877415 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:22.877332 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-575996c6dd-2tngr" Oct 30 16:58:22.953759 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:22.953714 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-lsdd7"] Oct 30 16:58:22.954073 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:22.954018 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" podUID="006b690d-277e-412c-b8c5-962660f560e3" containerName="manager" containerID="cri-o://9db1bfb28e6fcf31507297eac36c1ab485694607768c188bffd1c0f7a5ce2284" gracePeriod=30 Oct 30 16:58:23.214117 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.214091 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" Oct 30 16:58:23.229262 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.229238 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/006b690d-277e-412c-b8c5-962660f560e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/006b690d-277e-412c-b8c5-962660f560e3-metrics-certs\") pod \"006b690d-277e-412c-b8c5-962660f560e3\" (UID: \"006b690d-277e-412c-b8c5-962660f560e3\") " Oct 30 16:58:23.229352 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.229300 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006b690d-277e-412c-b8c5-962660f560e3-kube-api-access-5gxqq\" (UniqueName: \"kubernetes.io/projected/006b690d-277e-412c-b8c5-962660f560e3-kube-api-access-5gxqq\") pod \"006b690d-277e-412c-b8c5-962660f560e3\" (UID: \"006b690d-277e-412c-b8c5-962660f560e3\") " Oct 30 16:58:23.231302 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.231274 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/006b690d-277e-412c-b8c5-962660f560e3-metrics-certs" pod "006b690d-277e-412c-b8c5-962660f560e3" (UID: "006b690d-277e-412c-b8c5-962660f560e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:58:23.231423 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.231308 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006b690d-277e-412c-b8c5-962660f560e3-kube-api-access-5gxqq" pod "006b690d-277e-412c-b8c5-962660f560e3" (UID: "006b690d-277e-412c-b8c5-962660f560e3"). InnerVolumeSpecName "kube-api-access-5gxqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:23.330498 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.330466 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gxqq\" (UniqueName: \"kubernetes.io/projected/006b690d-277e-412c-b8c5-962660f560e3-kube-api-access-5gxqq\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:58:23.330498 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.330493 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006b690d-277e-412c-b8c5-962660f560e3-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 16:58:23.935677 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.935637 4828 generic.go:356] "Generic (PLEG): container finished" podID="006b690d-277e-412c-b8c5-962660f560e3" containerID="9db1bfb28e6fcf31507297eac36c1ab485694607768c188bffd1c0f7a5ce2284" exitCode=0 Oct 30 16:58:23.936107 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.935687 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" Oct 30 16:58:23.936107 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.935700 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" event={"ID":"006b690d-277e-412c-b8c5-962660f560e3","Type":"ContainerDied","Data":"9db1bfb28e6fcf31507297eac36c1ab485694607768c188bffd1c0f7a5ce2284"} Oct 30 16:58:23.936107 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.935752 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-lsdd7" event={"ID":"006b690d-277e-412c-b8c5-962660f560e3","Type":"ContainerDied","Data":"9c9b93b82aa28a66054df3f63f3f6822a240c0232cffef57cc3953277ea1bb47"} Oct 30 16:58:23.936107 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.935781 4828 scope.go:122] "RemoveContainer" containerID="9db1bfb28e6fcf31507297eac36c1ab485694607768c188bffd1c0f7a5ce2284" Oct 30 16:58:23.945639 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.945617 4828 scope.go:122] "RemoveContainer" containerID="9db1bfb28e6fcf31507297eac36c1ab485694607768c188bffd1c0f7a5ce2284" Oct 30 16:58:23.945883 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:58:23.945865 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db1bfb28e6fcf31507297eac36c1ab485694607768c188bffd1c0f7a5ce2284\": container with ID starting with 9db1bfb28e6fcf31507297eac36c1ab485694607768c188bffd1c0f7a5ce2284 not found: ID does not exist" containerID="9db1bfb28e6fcf31507297eac36c1ab485694607768c188bffd1c0f7a5ce2284" Oct 30 16:58:23.945967 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.945890 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db1bfb28e6fcf31507297eac36c1ab485694607768c188bffd1c0f7a5ce2284"} err="failed to get container status \"9db1bfb28e6fcf31507297eac36c1ab485694607768c188bffd1c0f7a5ce2284\": rpc error: code = NotFound desc = could not find container \"9db1bfb28e6fcf31507297eac36c1ab485694607768c188bffd1c0f7a5ce2284\": container with ID starting with 9db1bfb28e6fcf31507297eac36c1ab485694607768c188bffd1c0f7a5ce2284 not found: ID does not exist" Oct 30 16:58:23.991662 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:23.991626 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-lsdd7"] Oct 30 16:58:24.005010 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:24.004978 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-lsdd7"] Oct 30 16:58:24.494008 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:24.493973 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006b690d-277e-412c-b8c5-962660f560e3" path="/var/lib/kubelet/pods/006b690d-277e-412c-b8c5-962660f560e3/volumes" Oct 30 16:58:24.683118 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:24.683087 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:25.200026 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:25.199982 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:26.832314 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:26.832283 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:26.990659 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:26.990625 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:28.424868 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:28.424831 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:28.587308 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:28.587263 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:28.979850 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:28.979818 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:37.909598 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:58:37.909558 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:58:50.841147 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:50.841110 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:51.018798 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.018762 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:51.342663 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.342629 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll"] Oct 30 16:58:51.343114 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.343099 4828 cpu_manager.go:414] "containerMap: removing container" podUID="006b690d-277e-412c-b8c5-962660f560e3" containerName="manager" Oct 30 16:58:51.343173 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.343114 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006b690d-277e-412c-b8c5-962660f560e3" containerName="manager" Oct 30 16:58:51.343219 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.343207 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="006b690d-277e-412c-b8c5-962660f560e3" containerName="manager" Oct 30 16:58:51.350844 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.350820 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.378062 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.378031 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll"] Oct 30 16:58:51.388044 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.388008 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b80d83-openshift-default-dockercfg-6zl9l\"" Oct 30 16:58:51.438687 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.438648 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:51.480303 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.480268 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.480303 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.480307 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.480512 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.480326 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.480512 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.480481 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.480620 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.480510 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.480620 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.480532 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.480714 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.480688 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.480815 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.480798 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnmmw\" (UniqueName: \"kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-kube-api-access-fnmmw\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.480867 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.480841 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.498733 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.498702 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv"] Oct 30 16:58:51.502447 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.502419 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.520128 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.520093 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c7b9df-openshift-default-dockercfg-qhctr\"" Oct 30 16:58:51.531304 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.530633 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv"] Oct 30 16:58:51.581812 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.581776 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.582028 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.581842 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnmmw\" (UniqueName: \"kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-kube-api-access-fnmmw\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.582028 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.581926 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edde938-9f51-43c7-9411-128b417a3edc-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.582028 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.581957 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edde938-9f51-43c7-9411-128b417a3edc-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.582173 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582136 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.582173 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582161 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.582278 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582180 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.582278 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582247 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpgn\" (UniqueName: \"kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-kube-api-access-8lpgn\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.582394 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582289 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.582394 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582340 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.582394 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582366 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.582529 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582398 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.582529 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582454 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.582529 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582516 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.582642 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582529 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.582642 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582538 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.582642 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582596 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.582642 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582609 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.582834 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582717 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.582834 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582746 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.582834 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.582817 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.583130 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.583104 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.583261 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.583133 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.584869 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.584849 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.584869 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.584859 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.609196 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.609120 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.609325 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.609195 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnmmw\" (UniqueName: \"kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-kube-api-access-fnmmw\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.663447 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.663401 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:51.674118 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.674036 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:51.683945 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.683913 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.684209 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.683979 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.684209 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.684070 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.684209 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.684134 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.684209 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.684180 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edde938-9f51-43c7-9411-128b417a3edc-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.684209 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.684206 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edde938-9f51-43c7-9411-128b417a3edc-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.684790 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.684243 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.684790 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.684274 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.684790 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.684298 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpgn\" (UniqueName: \"kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-kube-api-access-8lpgn\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.684790 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.684409 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.684790 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.684688 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.686039 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.685511 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.686582 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.686288 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edde938-9f51-43c7-9411-128b417a3edc-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.686719 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.686514 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.689073 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.689025 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.690630 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.690606 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edde938-9f51-43c7-9411-128b417a3edc-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.716024 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.715958 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.742405 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.742377 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpgn\" (UniqueName: \"kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-kube-api-access-8lpgn\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.820545 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.820508 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:51.824961 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.824929 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj"] Oct 30 16:58:51.831706 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.831680 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.854665 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.854631 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b4f406-openshift-default-dockercfg-ftgfs\"" Oct 30 16:58:51.858017 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.857982 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:51.864300 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.864209 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj"] Oct 30 16:58:51.886835 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.886795 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.887552 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.887525 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.887739 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.887720 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.887929 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.887911 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.888051 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.888035 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.888204 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.888185 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.888333 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.888317 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.888465 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.888450 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.888624 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.888608 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64nl7\" (UniqueName: \"kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-kube-api-access-64nl7\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.989613 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.989578 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.989613 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.989616 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.989826 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.989654 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.989826 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.989684 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.989826 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.989710 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.989826 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.989736 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64nl7\" (UniqueName: \"kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-kube-api-access-64nl7\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.989826 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.989786 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.989826 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.989803 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.989826 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.989823 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.991033 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.990560 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.991033 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.990598 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.991033 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.990968 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.991517 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.991489 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.992213 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.992167 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.993732 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.993688 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:51.997253 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.997224 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll"] Oct 30 16:58:51.997595 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:51.997573 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:52.000315 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:58:51.999959 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a75163_e6e5_4eaa_9197_a0e16de86f19.slice/crio-0094ed4344b21270d28210d43197283aac3fda496eda16d3149fec231ecef089 WatchSource:0}: Error finding container 0094ed4344b21270d28210d43197283aac3fda496eda16d3149fec231ecef089: Status 404 returned error can't find the container with id 0094ed4344b21270d28210d43197283aac3fda496eda16d3149fec231ecef089 Oct 30 16:58:52.003108 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.003078 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.003184 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.003158 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.003229 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.003197 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.020472 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.020448 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:52.023031 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.023006 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64nl7\" (UniqueName: \"kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-kube-api-access-64nl7\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:52.031386 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.031260 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:52.088600 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.088567 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" event={"ID":"e2a75163-e6e5-4eaa-9197-a0e16de86f19","Type":"ContainerStarted","Data":"07db4d7706f33854aa2205387dbb67ba093f9bcc7f3e3a624319a67c9d0471ec"} Oct 30 16:58:52.088788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.088623 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" event={"ID":"e2a75163-e6e5-4eaa-9197-a0e16de86f19","Type":"ContainerStarted","Data":"0094ed4344b21270d28210d43197283aac3fda496eda16d3149fec231ecef089"} Oct 30 16:58:52.128000 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.127889 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq"] Oct 30 16:58:52.143256 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.143225 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.159613 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.155269 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:52.159998 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.159924 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq"] Oct 30 16:58:52.167673 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.167647 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-04248e-openshift-default-dockercfg-bbxjd\"" Oct 30 16:58:52.170600 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.169140 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" podStartSLOduration=1.169120586 podStartE2EDuration="1.169120586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:52.160047476 +0000 UTC m=+709.805557657" watchObservedRunningTime="2026-10-30 16:58:52.169120586 +0000 UTC m=+709.814630769" Oct 30 16:58:52.182069 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.180941 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv"] Oct 30 16:58:52.209572 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:58:52.209537 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edde938_9f51_43c7_9411_128b417a3edc.slice/crio-daed19503e01908629f5dd37acdb9eded18b39faf03fabd27df194c0b8936947 WatchSource:0}: Error finding container daed19503e01908629f5dd37acdb9eded18b39faf03fabd27df194c0b8936947: Status 404 returned error can't find the container with id daed19503e01908629f5dd37acdb9eded18b39faf03fabd27df194c0b8936947 Oct 30 16:58:52.215276 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.215243 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:52.217044 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.216997 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.217146 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.217105 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.217212 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.217178 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.303388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.302824 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s67sd\" (UniqueName: \"kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-kube-api-access-s67sd\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.303388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.302875 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.303388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.302942 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.303388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.302971 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.303388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.303001 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.303388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.303050 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05de1e06-0e9b-46a5-9020-5d791da7654a-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.303388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.303089 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.303388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.303131 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.303388 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.303180 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.378252 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.378023 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:52.404481 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.404442 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.404691 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.404528 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s67sd\" (UniqueName: \"kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-kube-api-access-s67sd\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.404691 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.404575 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.404691 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.404620 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.404691 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.404648 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.404691 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.404676 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.405018 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.404719 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05de1e06-0e9b-46a5-9020-5d791da7654a-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.405018 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.404758 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.405018 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.404798 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.405595 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.405567 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.405994 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.405968 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.406565 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.406534 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.406663 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.406572 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.407164 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.407138 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05de1e06-0e9b-46a5-9020-5d791da7654a-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.407580 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.407555 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.409585 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.409542 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.431781 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.431730 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.431781 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.431743 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s67sd\" (UniqueName: \"kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-kube-api-access-s67sd\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.532944 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.518134 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:52.536515 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.536482 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:52.553720 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.553684 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj"] Oct 30 16:58:52.564982 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:58:52.564942 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36000b4a_d11f_483e_9a73_e5bdf70daa5a.slice/crio-28b32f9903441bb7e4a20f1f707cfabd3de3a4a84917657ec3f747918f99cd10 WatchSource:0}: Error finding container 28b32f9903441bb7e4a20f1f707cfabd3de3a4a84917657ec3f747918f99cd10: Status 404 returned error can't find the container with id 28b32f9903441bb7e4a20f1f707cfabd3de3a4a84917657ec3f747918f99cd10 Oct 30 16:58:52.567525 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.567488 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.567645 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.567576 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.567645 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.567622 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.664367 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.664331 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:52.891006 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:58:52.890975 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05de1e06_0e9b_46a5_9020_5d791da7654a.slice/crio-7b29e02c4e68f008e96b08116e9b44722cf07aa2b2823eaf840d4dd855438d44 WatchSource:0}: Error finding container 7b29e02c4e68f008e96b08116e9b44722cf07aa2b2823eaf840d4dd855438d44: Status 404 returned error can't find the container with id 7b29e02c4e68f008e96b08116e9b44722cf07aa2b2823eaf840d4dd855438d44 Oct 30 16:58:52.891996 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.891969 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq"] Oct 30 16:58:52.893620 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.893581 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.893700 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.893679 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:52.893741 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:52.893722 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:58:53.095789 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.095752 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" event={"ID":"36000b4a-d11f-483e-9a73-e5bdf70daa5a","Type":"ContainerStarted","Data":"a1381e78acbc22a2e8a2803e9bdc59c4ff74442edff9893a30ef23fbf2dc00e1"} Oct 30 16:58:53.096004 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.095820 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" event={"ID":"36000b4a-d11f-483e-9a73-e5bdf70daa5a","Type":"ContainerStarted","Data":"28b32f9903441bb7e4a20f1f707cfabd3de3a4a84917657ec3f747918f99cd10"} Oct 30 16:58:53.097454 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.097427 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" event={"ID":"9edde938-9f51-43c7-9411-128b417a3edc","Type":"ContainerStarted","Data":"ed356183a53424b1347f9f31609e026763d711c1eb9fc71083ccac7f09c22347"} Oct 30 16:58:53.097572 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.097479 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" event={"ID":"9edde938-9f51-43c7-9411-128b417a3edc","Type":"ContainerStarted","Data":"daed19503e01908629f5dd37acdb9eded18b39faf03fabd27df194c0b8936947"} Oct 30 16:58:53.099370 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.099344 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" event={"ID":"05de1e06-0e9b-46a5-9020-5d791da7654a","Type":"ContainerStarted","Data":"7e7972e3031c2ce34e5ec3e5b93f502d89aeb3035d4a4844d0123ba157569972"} Oct 30 16:58:53.099481 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.099391 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" event={"ID":"05de1e06-0e9b-46a5-9020-5d791da7654a","Type":"ContainerStarted","Data":"7b29e02c4e68f008e96b08116e9b44722cf07aa2b2823eaf840d4dd855438d44"} Oct 30 16:58:53.145754 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.145620 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" podStartSLOduration=2.145603509 podStartE2EDuration="2.145603509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:53.143886291 +0000 UTC m=+710.789396474" watchObservedRunningTime="2026-10-30 16:58:53.145603509 +0000 UTC m=+710.791113691" Oct 30 16:58:53.156138 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.156098 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:53.162424 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.162303 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:53.198473 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.198399 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" podStartSLOduration=1.198377012 podStartE2EDuration="1.198377012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:53.195812795 +0000 UTC m=+710.841322976" watchObservedRunningTime="2026-10-30 16:58:53.198377012 +0000 UTC m=+710.843887193" Oct 30 16:58:53.246092 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.246040 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" podStartSLOduration=2.246021455 podStartE2EDuration="2.246021455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:53.243607854 +0000 UTC m=+710.889118062" watchObservedRunningTime="2026-10-30 16:58:53.246021455 +0000 UTC m=+710.891531635" Oct 30 16:58:53.521734 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.521705 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:53.526867 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.526845 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:53.671994 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.671964 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:53.754068 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.754035 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:53.821004 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.820919 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:53.826026 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:53.826000 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:54.105122 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:54.105040 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:54.105550 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:54.105129 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:54.105550 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:54.105157 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:54.105550 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:54.105180 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:54.105707 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:54.105579 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:58:54.105707 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:54.105651 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:58:54.105707 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:54.105695 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:58:54.106436 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:54.106414 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:58:55.135740 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:55.135696 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:55.315983 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:55.315940 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:55.831357 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:55.831322 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:56.077725 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:56.077689 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:56.281930 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:56.281862 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:56.528407 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:56.528367 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:56.720352 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:56.720268 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:56.881995 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:56.881957 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:57.120103 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:57.120064 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:57.287861 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:57.287824 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:57.890634 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:57.890597 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:58.042705 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.042661 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:58.052528 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.052495 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl"] Oct 30 16:58:58.057559 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.057540 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" Oct 30 16:58:58.080950 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.078982 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl"] Oct 30 16:58:58.088672 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.088638 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b80d83\"" Oct 30 16:58:58.088832 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.088749 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-b80d83-dockercfg-2m7lr\"" Oct 30 16:58:58.178690 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.178587 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8b6c\" (UniqueName: \"kubernetes.io/projected/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-kube-api-access-z8b6c\") pod \"payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl\" (UID: \"c4eb9078-46ec-4fa4-b0a6-dc78e060859e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" Oct 30 16:58:58.178863 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.178729 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl\" (UID: \"c4eb9078-46ec-4fa4-b0a6-dc78e060859e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" Oct 30 16:58:58.279303 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.279265 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl\" (UID: \"c4eb9078-46ec-4fa4-b0a6-dc78e060859e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" Oct 30 16:58:58.279521 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.279489 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8b6c\" (UniqueName: \"kubernetes.io/projected/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-kube-api-access-z8b6c\") pod \"payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl\" (UID: \"c4eb9078-46ec-4fa4-b0a6-dc78e060859e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" Oct 30 16:58:58.280092 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.280065 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl\" (UID: \"c4eb9078-46ec-4fa4-b0a6-dc78e060859e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" Oct 30 16:58:58.304467 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.304440 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8b6c\" (UniqueName: \"kubernetes.io/projected/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-kube-api-access-z8b6c\") pod \"payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl\" (UID: \"c4eb9078-46ec-4fa4-b0a6-dc78e060859e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" Oct 30 16:58:58.370666 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.370614 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" Oct 30 16:58:58.678676 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.678644 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl"] Oct 30 16:58:58.679168 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:58:58.679095 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4eb9078_46ec_4fa4_b0a6_dc78e060859e.slice/crio-2bb6cfb2c75858b72bc4147cb82d3f48a7c08a13e729086921e834a07b3b2eb6 WatchSource:0}: Error finding container 2bb6cfb2c75858b72bc4147cb82d3f48a7c08a13e729086921e834a07b3b2eb6: Status 404 returned error can't find the container with id 2bb6cfb2c75858b72bc4147cb82d3f48a7c08a13e729086921e834a07b3b2eb6 Oct 30 16:58:58.927714 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:58.927675 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:59.068812 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.068776 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh"] Oct 30 16:58:59.072827 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.072803 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" Oct 30 16:58:59.080067 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.079978 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:59.091846 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.091823 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-86f50a\"" Oct 30 16:58:59.091969 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.091912 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-86f50a-dockercfg-bxmtq\"" Oct 30 16:58:59.094770 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.094744 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh"] Oct 30 16:58:59.186816 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.186784 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf"] Oct 30 16:58:59.191636 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.191606 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh\" (UID: \"14d8d98a-d3b2-4c79-9ebe-d9f817498f61\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" Oct 30 16:58:59.191832 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.191708 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99fmh\" (UniqueName: \"kubernetes.io/projected/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-kube-api-access-99fmh\") pod \"payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh\" (UID: \"14d8d98a-d3b2-4c79-9ebe-d9f817498f61\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" Oct 30 16:58:59.194644 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.194614 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" Oct 30 16:58:59.194811 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.194656 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" event={"ID":"c4eb9078-46ec-4fa4-b0a6-dc78e060859e","Type":"ContainerStarted","Data":"daeca9c816e887a99e2adf62b95eefb00178099c5e06b1c4ee607eeae761c4b1"} Oct 30 16:58:59.194811 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.194682 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" event={"ID":"c4eb9078-46ec-4fa4-b0a6-dc78e060859e","Type":"ContainerStarted","Data":"2bb6cfb2c75858b72bc4147cb82d3f48a7c08a13e729086921e834a07b3b2eb6"} Oct 30 16:58:59.194811 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.194723 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" Oct 30 16:58:59.200426 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.200375 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf"] Oct 30 16:58:59.240150 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.240097 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" podStartSLOduration=1.240083437 podStartE2EDuration="1.240083437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:59.236987005 +0000 UTC m=+716.882497200" watchObservedRunningTime="2026-10-30 16:58:59.240083437 +0000 UTC m=+716.885593614" Oct 30 16:58:59.293252 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.293217 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666617d1-e514-4b17-ae65-1da9dfeea316-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf\" (UID: \"666617d1-e514-4b17-ae65-1da9dfeea316\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" Oct 30 16:58:59.293418 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.293283 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh\" (UID: \"14d8d98a-d3b2-4c79-9ebe-d9f817498f61\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" Oct 30 16:58:59.293418 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.293359 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8cnh\" (UniqueName: \"kubernetes.io/projected/666617d1-e514-4b17-ae65-1da9dfeea316-kube-api-access-m8cnh\") pod \"payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf\" (UID: \"666617d1-e514-4b17-ae65-1da9dfeea316\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" Oct 30 16:58:59.293505 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.293431 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99fmh\" (UniqueName: \"kubernetes.io/projected/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-kube-api-access-99fmh\") pod \"payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh\" (UID: \"14d8d98a-d3b2-4c79-9ebe-d9f817498f61\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" Oct 30 16:58:59.294072 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.294053 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh\" (UID: \"14d8d98a-d3b2-4c79-9ebe-d9f817498f61\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" Oct 30 16:58:59.318397 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.318371 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99fmh\" (UniqueName: \"kubernetes.io/projected/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-kube-api-access-99fmh\") pod \"payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh\" (UID: \"14d8d98a-d3b2-4c79-9ebe-d9f817498f61\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" Oct 30 16:58:59.386417 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.386330 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" Oct 30 16:58:59.394956 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.394925 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8cnh\" (UniqueName: \"kubernetes.io/projected/666617d1-e514-4b17-ae65-1da9dfeea316-kube-api-access-m8cnh\") pod \"payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf\" (UID: \"666617d1-e514-4b17-ae65-1da9dfeea316\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" Oct 30 16:58:59.395060 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.395034 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666617d1-e514-4b17-ae65-1da9dfeea316-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf\" (UID: \"666617d1-e514-4b17-ae65-1da9dfeea316\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" Oct 30 16:58:59.395719 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.395698 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666617d1-e514-4b17-ae65-1da9dfeea316-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf\" (UID: \"666617d1-e514-4b17-ae65-1da9dfeea316\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" Oct 30 16:58:59.419808 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.419771 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8cnh\" (UniqueName: \"kubernetes.io/projected/666617d1-e514-4b17-ae65-1da9dfeea316-kube-api-access-m8cnh\") pod \"payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf\" (UID: \"666617d1-e514-4b17-ae65-1da9dfeea316\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" Oct 30 16:58:59.515004 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.514964 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" Oct 30 16:58:59.692520 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.692486 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh"] Oct 30 16:58:59.693367 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:58:59.693342 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d8d98a_d3b2_4c79_9ebe_d9f817498f61.slice/crio-1a51d94c1f8e64dd25288db9323354aa68b7852115507ae481008a7ca332630b WatchSource:0}: Error finding container 1a51d94c1f8e64dd25288db9323354aa68b7852115507ae481008a7ca332630b: Status 404 returned error can't find the container with id 1a51d94c1f8e64dd25288db9323354aa68b7852115507ae481008a7ca332630b Oct 30 16:58:59.822295 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.822268 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf"] Oct 30 16:58:59.828416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.828388 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:58:59.957853 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.957773 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45"] Oct 30 16:58:59.961694 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.961667 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" Oct 30 16:58:59.980232 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.980201 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-096d75\"" Oct 30 16:58:59.980507 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.980485 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-096d75-dockercfg-q7rzx\"" Oct 30 16:58:59.983358 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.983335 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45"] Oct 30 16:58:59.985982 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:58:59.985957 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:00.103697 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.103666 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf528e56-af27-46d6-8f3d-20cc44101152-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45\" (UID: \"cf528e56-af27-46d6-8f3d-20cc44101152\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" Oct 30 16:59:00.103918 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.103878 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdph2\" (UniqueName: \"kubernetes.io/projected/cf528e56-af27-46d6-8f3d-20cc44101152-kube-api-access-hdph2\") pod \"payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45\" (UID: \"cf528e56-af27-46d6-8f3d-20cc44101152\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" Oct 30 16:59:00.201276 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.201234 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" event={"ID":"14d8d98a-d3b2-4c79-9ebe-d9f817498f61","Type":"ContainerStarted","Data":"336b6335ef8493e9436208e409aca31f68e807b9e164599ede7bc7e7e1bf06aa"} Oct 30 16:59:00.201447 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.201313 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" event={"ID":"14d8d98a-d3b2-4c79-9ebe-d9f817498f61","Type":"ContainerStarted","Data":"1a51d94c1f8e64dd25288db9323354aa68b7852115507ae481008a7ca332630b"} Oct 30 16:59:00.201447 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.201345 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" Oct 30 16:59:00.203101 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.203072 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" event={"ID":"666617d1-e514-4b17-ae65-1da9dfeea316","Type":"ContainerStarted","Data":"234edcfaa0fd28d71f3e68f93d2456a12083e270067bf98e5d652e0ca18c9588"} Oct 30 16:59:00.203202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.203122 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" event={"ID":"666617d1-e514-4b17-ae65-1da9dfeea316","Type":"ContainerStarted","Data":"cbe24a0395b5b6f80d7348809915e02aa295422a857575b6b75dbface0571404"} Oct 30 16:59:00.203202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.203172 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" Oct 30 16:59:00.204708 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.204692 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf528e56-af27-46d6-8f3d-20cc44101152-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45\" (UID: \"cf528e56-af27-46d6-8f3d-20cc44101152\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" Oct 30 16:59:00.204788 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.204775 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdph2\" (UniqueName: \"kubernetes.io/projected/cf528e56-af27-46d6-8f3d-20cc44101152-kube-api-access-hdph2\") pod \"payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45\" (UID: \"cf528e56-af27-46d6-8f3d-20cc44101152\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" Oct 30 16:59:00.205378 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.205360 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf528e56-af27-46d6-8f3d-20cc44101152-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45\" (UID: \"cf528e56-af27-46d6-8f3d-20cc44101152\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" Oct 30 16:59:00.230342 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.230273 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdph2\" (UniqueName: \"kubernetes.io/projected/cf528e56-af27-46d6-8f3d-20cc44101152-kube-api-access-hdph2\") pod \"payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45\" (UID: \"cf528e56-af27-46d6-8f3d-20cc44101152\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" Oct 30 16:59:00.249098 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.249047 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" podStartSLOduration=1.249034482 podStartE2EDuration="1.249034482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:00.248296524 +0000 UTC m=+717.893806703" watchObservedRunningTime="2026-10-30 16:59:00.249034482 +0000 UTC m=+717.894544662" Oct 30 16:59:00.275151 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.275117 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" Oct 30 16:59:00.300794 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.300768 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr"] Oct 30 16:59:00.302696 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.302637 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" podStartSLOduration=1.302619425 podStartE2EDuration="1.302619425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:00.300548256 +0000 UTC m=+717.946058436" watchObservedRunningTime="2026-10-30 16:59:00.302619425 +0000 UTC m=+717.948129603" Oct 30 16:59:00.310055 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.310031 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:00.332938 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.332885 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr"] Oct 30 16:59:00.344133 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.344103 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-b80d83-serving-cert\"" Oct 30 16:59:00.429223 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.429172 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:00.508365 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.508331 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-metrics-certs\") pod \"maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:00.508560 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.508401 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:00.508560 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.508442 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:00.509281 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.508728 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm27h\" (UniqueName: \"kubernetes.io/projected/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-kube-api-access-fm27h\") pod \"maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:00.583202 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.583159 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:00.599511 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.599483 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45"] Oct 30 16:59:00.600049 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:59:00.600011 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf528e56_af27_46d6_8f3d_20cc44101152.slice/crio-90c1acd243e0f2d24943c7b8d41768afff8d235834eeac9b5f51c9e01c534d6e WatchSource:0}: Error finding container 90c1acd243e0f2d24943c7b8d41768afff8d235834eeac9b5f51c9e01c534d6e: Status 404 returned error can't find the container with id 90c1acd243e0f2d24943c7b8d41768afff8d235834eeac9b5f51c9e01c534d6e Oct 30 16:59:00.609864 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.609838 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm27h\" (UniqueName: \"kubernetes.io/projected/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-kube-api-access-fm27h\") pod \"maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:00.610003 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.609929 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-metrics-certs\") pod \"maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:00.610003 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.609988 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:00.610135 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.610027 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:00.610197 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:59:00.610125 4828 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-serving-cert: secret "maas-api-e2e-worker-w3-b80d83-serving-cert" not found Oct 30 16:59:00.610251 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:59:00.610214 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-maas-api-tls podName:606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a nodeName:}" failed. No retries permitted until 2026-10-30 16:59:01.110179426 +0000 UTC m=+718.755689597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-maas-api-tls") pod "maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" (UID: "606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a") : secret "maas-api-e2e-worker-w3-b80d83-serving-cert" not found Oct 30 16:59:00.610766 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.610743 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:00.612721 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.612698 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-metrics-certs\") pod \"maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:00.634916 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.634872 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm27h\" (UniqueName: \"kubernetes.io/projected/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-kube-api-access-fm27h\") pod \"maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:00.799625 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.799533 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:00.948783 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:00.948741 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:01.115247 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.115154 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:01.117954 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.117923 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:01.210916 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.210864 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" event={"ID":"cf528e56-af27-46d6-8f3d-20cc44101152","Type":"ContainerStarted","Data":"d4bb4bd6bc1d280252c085b76de47cb11c2c372c86dee8e90827733b949c97d4"} Oct 30 16:59:01.211106 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.210943 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" event={"ID":"cf528e56-af27-46d6-8f3d-20cc44101152","Type":"ContainerStarted","Data":"90c1acd243e0f2d24943c7b8d41768afff8d235834eeac9b5f51c9e01c534d6e"} Oct 30 16:59:01.211106 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.210978 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" Oct 30 16:59:01.249038 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.249005 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:01.257391 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.257330 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" podStartSLOduration=2.257311621 podStartE2EDuration="2.257311621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:01.253878316 +0000 UTC m=+718.899388495" watchObservedRunningTime="2026-10-30 16:59:01.257311621 +0000 UTC m=+718.902821801" Oct 30 16:59:01.566383 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.566345 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr"] Oct 30 16:59:01.566986 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:59:01.566957 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606ccbcf_d73f_4e1e_9dc2_63b217fdcf2a.slice/crio-a4e7bc69a4c7838b990dc655b25ad909fab0c35b925990431cb73f691525460b WatchSource:0}: Error finding container a4e7bc69a4c7838b990dc655b25ad909fab0c35b925990431cb73f691525460b: Status 404 returned error can't find the container with id a4e7bc69a4c7838b990dc655b25ad909fab0c35b925990431cb73f691525460b Oct 30 16:59:01.766908 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.766857 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:01.893928 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.893837 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h"] Oct 30 16:59:01.899070 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.899050 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" Oct 30 16:59:01.916607 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.916578 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-d384dc\"" Oct 30 16:59:01.929167 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.929136 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-d384dc-dockercfg-fl7bd\"" Oct 30 16:59:01.929368 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.929285 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:01.932523 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:01.932499 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h"] Oct 30 16:59:02.030458 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.030401 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8fmw\" (UniqueName: \"kubernetes.io/projected/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-kube-api-access-g8fmw\") pod \"payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h\" (UID: \"6eb2dd00-1748-4fee-a0c5-1a600bed0c18\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" Oct 30 16:59:02.030458 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.030455 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h\" (UID: \"6eb2dd00-1748-4fee-a0c5-1a600bed0c18\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" Oct 30 16:59:02.131555 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.131515 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8fmw\" (UniqueName: \"kubernetes.io/projected/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-kube-api-access-g8fmw\") pod \"payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h\" (UID: \"6eb2dd00-1748-4fee-a0c5-1a600bed0c18\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" Oct 30 16:59:02.131555 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.131561 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h\" (UID: \"6eb2dd00-1748-4fee-a0c5-1a600bed0c18\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" Oct 30 16:59:02.132297 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.132276 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h\" (UID: \"6eb2dd00-1748-4fee-a0c5-1a600bed0c18\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" Oct 30 16:59:02.157033 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.156965 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8fmw\" (UniqueName: \"kubernetes.io/projected/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-kube-api-access-g8fmw\") pod \"payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h\" (UID: \"6eb2dd00-1748-4fee-a0c5-1a600bed0c18\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" Oct 30 16:59:02.213799 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.213743 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" Oct 30 16:59:02.219434 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.219402 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" event={"ID":"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a","Type":"ContainerStarted","Data":"683d84eaf6a8a25c30f7a1846905a59cc8cdfe339a2f71bd05ff641a0fd9bdd1"} Oct 30 16:59:02.219554 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.219461 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" event={"ID":"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a","Type":"ContainerStarted","Data":"a4e7bc69a4c7838b990dc655b25ad909fab0c35b925990431cb73f691525460b"} Oct 30 16:59:02.219818 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.219795 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:02.266583 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.266522 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" podStartSLOduration=1.8815319499999998 podStartE2EDuration="2.266502187s" totalImagesPullingTime="384.970237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:02.264994952 +0000 UTC m=+719.910505133" watchObservedRunningTime="2026-10-30 16:59:02.266502187 +0000 UTC m=+719.912012367" Oct 30 16:59:02.392541 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.392501 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7"] Oct 30 16:59:02.399817 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.399788 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.414256 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.414223 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7"] Oct 30 16:59:02.506090 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.506061 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-86f50a-serving-cert\"" Oct 30 16:59:02.514895 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.514859 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:02.519358 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.519333 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h"] Oct 30 16:59:02.519571 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:59:02.519550 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb2dd00_1748_4fee_a0c5_1a600bed0c18.slice/crio-4ef77522db7db95f57fd0ee7ab2211142515bdd529eeb0e4252fe72d8b1c8a01 WatchSource:0}: Error finding container 4ef77522db7db95f57fd0ee7ab2211142515bdd529eeb0e4252fe72d8b1c8a01: Status 404 returned error can't find the container with id 4ef77522db7db95f57fd0ee7ab2211142515bdd529eeb0e4252fe72d8b1c8a01 Oct 30 16:59:02.547343 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.547309 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07e31526-445e-4541-8026-e134539da213-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.547501 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.547480 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-metrics-certs\") pod \"maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.547617 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.547603 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-maas-api-tls\") pod \"maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.547653 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.547633 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k9nf\" (UniqueName: \"kubernetes.io/projected/07e31526-445e-4541-8026-e134539da213-kube-api-access-7k9nf\") pod \"maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.654017 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.653787 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07e31526-445e-4541-8026-e134539da213-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.656192 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.654460 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-metrics-certs\") pod \"maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.656192 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.654575 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-maas-api-tls\") pod \"maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.656192 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.654611 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k9nf\" (UniqueName: \"kubernetes.io/projected/07e31526-445e-4541-8026-e134539da213-kube-api-access-7k9nf\") pod \"maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.656192 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.655716 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07e31526-445e-4541-8026-e134539da213-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.667679 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.667613 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-metrics-certs\") pod \"maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.669909 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.669830 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-maas-api-tls\") pod \"maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.680223 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.680100 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k9nf\" (UniqueName: \"kubernetes.io/projected/07e31526-445e-4541-8026-e134539da213-kube-api-access-7k9nf\") pod \"maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.703048 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.703016 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:02.716942 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.716909 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:02.882972 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.882884 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:02.981368 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:02.981283 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js"] Oct 30 16:59:03.011318 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.011279 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js"] Oct 30 16:59:03.011496 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.011418 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" Oct 30 16:59:03.027685 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.027653 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-04248e-dockercfg-82zpn\"" Oct 30 16:59:03.043341 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:59:03.043314 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e31526_445e_4541_8026_e134539da213.slice/crio-d69886c8f3358a0c2316c8f62e5e384c9fc98adf516363bfadd5ce09027fa1e8 WatchSource:0}: Error finding container d69886c8f3358a0c2316c8f62e5e384c9fc98adf516363bfadd5ce09027fa1e8: Status 404 returned error can't find the container with id d69886c8f3358a0c2316c8f62e5e384c9fc98adf516363bfadd5ce09027fa1e8 Oct 30 16:59:03.044489 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.044467 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7"] Oct 30 16:59:03.058878 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.058850 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1dfb448-5107-40ae-a8a7-6766274ad9ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js\" (UID: \"b1dfb448-5107-40ae-a8a7-6766274ad9ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" Oct 30 16:59:03.059088 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.059069 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9f8\" (UniqueName: \"kubernetes.io/projected/b1dfb448-5107-40ae-a8a7-6766274ad9ce-kube-api-access-9b9f8\") pod \"payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js\" (UID: \"b1dfb448-5107-40ae-a8a7-6766274ad9ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" Oct 30 16:59:03.081699 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.081671 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-04248e\"" Oct 30 16:59:03.160403 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.160361 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9f8\" (UniqueName: \"kubernetes.io/projected/b1dfb448-5107-40ae-a8a7-6766274ad9ce-kube-api-access-9b9f8\") pod \"payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js\" (UID: \"b1dfb448-5107-40ae-a8a7-6766274ad9ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" Oct 30 16:59:03.160403 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.160404 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1dfb448-5107-40ae-a8a7-6766274ad9ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js\" (UID: \"b1dfb448-5107-40ae-a8a7-6766274ad9ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" Oct 30 16:59:03.161351 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.161331 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1dfb448-5107-40ae-a8a7-6766274ad9ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js\" (UID: \"b1dfb448-5107-40ae-a8a7-6766274ad9ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" Oct 30 16:59:03.190130 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.190098 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9f8\" (UniqueName: \"kubernetes.io/projected/b1dfb448-5107-40ae-a8a7-6766274ad9ce-kube-api-access-9b9f8\") pod \"payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js\" (UID: \"b1dfb448-5107-40ae-a8a7-6766274ad9ce\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" Oct 30 16:59:03.226957 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.226919 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" event={"ID":"07e31526-445e-4541-8026-e134539da213","Type":"ContainerStarted","Data":"d69886c8f3358a0c2316c8f62e5e384c9fc98adf516363bfadd5ce09027fa1e8"} Oct 30 16:59:03.228628 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.228597 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" event={"ID":"6eb2dd00-1748-4fee-a0c5-1a600bed0c18","Type":"ContainerStarted","Data":"7bd635e7b6ca3cf218a3ab2284bf9cd718a46404fdd028269a49052dc72362eb"} Oct 30 16:59:03.228798 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.228763 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" Oct 30 16:59:03.228868 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.228797 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" event={"ID":"6eb2dd00-1748-4fee-a0c5-1a600bed0c18","Type":"ContainerStarted","Data":"4ef77522db7db95f57fd0ee7ab2211142515bdd529eeb0e4252fe72d8b1c8a01"} Oct 30 16:59:03.291421 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.291364 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" podStartSLOduration=2.291351161 podStartE2EDuration="2.291351161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:03.287741302 +0000 UTC m=+720.933251494" watchObservedRunningTime="2026-10-30 16:59:03.291351161 +0000 UTC m=+720.936861340" Oct 30 16:59:03.327959 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.327913 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" Oct 30 16:59:03.647593 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.647541 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js"] Oct 30 16:59:03.719656 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.719626 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:03.889178 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:03.888875 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:04.237020 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.236926 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" event={"ID":"b1dfb448-5107-40ae-a8a7-6766274ad9ce","Type":"ContainerStarted","Data":"a7013b65c85d127e323cc94864ba063869fc56618176f3432a6353aea62e59f3"} Oct 30 16:59:04.237020 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.236986 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" event={"ID":"b1dfb448-5107-40ae-a8a7-6766274ad9ce","Type":"ContainerStarted","Data":"7e2007462eb681df783c95062c485f45cf7dc9d15fc09df91e70917798edc1a4"} Oct 30 16:59:04.237255 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.237047 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" Oct 30 16:59:04.238609 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.238579 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" event={"ID":"07e31526-445e-4541-8026-e134539da213","Type":"ContainerStarted","Data":"8c8cc74f33cfb7c233827b75d01d7961b691dd345022d5829f4d72a3b65bdb0e"} Oct 30 16:59:04.238723 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.238682 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:04.284566 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.284531 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp"] Oct 30 16:59:04.295630 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.293361 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.299128 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.299066 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" podStartSLOduration=2.299047959 podStartE2EDuration="2.299047959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:04.293000762 +0000 UTC m=+721.938510943" watchObservedRunningTime="2026-10-30 16:59:04.299047959 +0000 UTC m=+721.944558189" Oct 30 16:59:04.314011 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.313460 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp"] Oct 30 16:59:04.314513 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.314489 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-096d75-serving-cert\"" Oct 30 16:59:04.349740 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.349678 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" podStartSLOduration=1.879164448 podStartE2EDuration="2.349658939s" totalImagesPullingTime="470.494491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:04.347225154 +0000 UTC m=+721.992735333" watchObservedRunningTime="2026-10-30 16:59:04.349658939 +0000 UTC m=+721.995169120" Oct 30 16:59:04.396924 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.396476 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-metrics-certs\") pod \"maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.396924 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.396571 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtchj\" (UniqueName: \"kubernetes.io/projected/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-kube-api-access-qtchj\") pod \"maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.396924 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.396640 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.396924 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.396745 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-maas-api-tls\") pod \"maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.429541 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.429498 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:04.500644 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.500547 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-maas-api-tls\") pod \"maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.500965 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.500938 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-metrics-certs\") pod \"maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.501217 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.501197 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtchj\" (UniqueName: \"kubernetes.io/projected/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-kube-api-access-qtchj\") pod \"maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.501402 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.501384 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.502365 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.502339 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.506188 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.506159 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-maas-api-tls\") pod \"maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.506678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.506658 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-metrics-certs\") pod \"maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.528356 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.528330 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtchj\" (UniqueName: \"kubernetes.io/projected/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-kube-api-access-qtchj\") pod \"maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.594680 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.594553 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:04.657005 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.656973 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:04.997319 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:04.997287 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp"] Oct 30 16:59:04.999412 ip-10-0-135-153 kubenswrapper[4828]: W1030 16:59:04.999386 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f0f53d_5f1a_4068_bea3_185bb56ab9ff.slice/crio-e8ec082109be36e42d9bfad917cfe8a3dceb13669c3156060ca241c478f5da3d WatchSource:0}: Error finding container e8ec082109be36e42d9bfad917cfe8a3dceb13669c3156060ca241c478f5da3d: Status 404 returned error can't find the container with id e8ec082109be36e42d9bfad917cfe8a3dceb13669c3156060ca241c478f5da3d Oct 30 16:59:05.245290 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:05.245250 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" event={"ID":"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff","Type":"ContainerStarted","Data":"e8ec082109be36e42d9bfad917cfe8a3dceb13669c3156060ca241c478f5da3d"} Oct 30 16:59:06.237674 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:06.237630 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:06.264965 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:06.264928 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" event={"ID":"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff","Type":"ContainerStarted","Data":"85ae1bb8a04c850d5d0f8794619b2ba957c4026d5cdebed9724e5be91aae263f"} Oct 30 16:59:06.265140 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:06.265006 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:06.312237 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:06.312163 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" podStartSLOduration=1.652950122 podStartE2EDuration="2.312143883s" totalImagesPullingTime="659.193761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:06.310487401 +0000 UTC m=+723.955997588" watchObservedRunningTime="2026-10-30 16:59:06.312143883 +0000 UTC m=+723.957654065" Oct 30 16:59:07.337802 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:07.337762 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:07.899924 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:07.899217 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h"] Oct 30 16:59:07.903978 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:07.903945 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:07.924114 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:07.924082 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h"] Oct 30 16:59:07.924861 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:07.924835 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-b4f406-serving-cert\"" Oct 30 16:59:07.936241 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:59:07.936092 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:59:07.944074 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:07.944038 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-metrics-certs\") pod \"maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:07.944411 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:07.944383 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bda2af-c577-42eb-bfdf-1b63d8433886-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:07.944519 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:07.944470 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:07.944579 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:07.944503 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97tg6\" (UniqueName: \"kubernetes.io/projected/66bda2af-c577-42eb-bfdf-1b63d8433886-kube-api-access-97tg6\") pod \"maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:08.015546 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:08.015507 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:08.045407 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:08.045372 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bda2af-c577-42eb-bfdf-1b63d8433886-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:08.045596 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:08.045424 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:08.045596 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:08.045451 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97tg6\" (UniqueName: \"kubernetes.io/projected/66bda2af-c577-42eb-bfdf-1b63d8433886-kube-api-access-97tg6\") pod \"maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:08.045596 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:08.045496 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-metrics-certs\") pod \"maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:08.046156 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:08.046134 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bda2af-c577-42eb-bfdf-1b63d8433886-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:08.047972 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:08.047954 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-metrics-certs\") pod \"maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:08.048309 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:08.048294 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:08.130535 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:08.130500 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97tg6\" (UniqueName: \"kubernetes.io/projected/66bda2af-c577-42eb-bfdf-1b63d8433886-kube-api-access-97tg6\") pod \"maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:08.228221 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:08.228135 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:08.233810 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:08.233781 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 16:59:08.234557 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:08.234542 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:08.605848 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:08.601354 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h"] Oct 30 16:59:09.335179 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:09.335091 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" event={"ID":"66bda2af-c577-42eb-bfdf-1b63d8433886","Type":"ContainerStarted","Data":"f773a2670898810e407ad7c8ee4cfe382c504d42eff177727a06f5d7f5034c92"} Oct 30 16:59:09.335179 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:09.335167 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" event={"ID":"66bda2af-c577-42eb-bfdf-1b63d8433886","Type":"ContainerStarted","Data":"e464b906497cd20708afd5485ccf63b5dfc888fac347aaa71b852fbe01da6f35"} Oct 30 16:59:09.335391 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:09.335243 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:09.388925 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:09.385436 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" podStartSLOduration=1.9460582180000001 podStartE2EDuration="2.385416803s" totalImagesPullingTime="439.358585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:09.383332061 +0000 UTC m=+727.028842243" watchObservedRunningTime="2026-10-30 16:59:09.385416803 +0000 UTC m=+727.030926985" Oct 30 16:59:09.966028 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:09.965991 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:10.115021 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:10.114977 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:10.206507 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:10.206481 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" Oct 30 16:59:10.251756 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:10.251688 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 16:59:11.215106 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.215069 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" Oct 30 16:59:11.215505 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.215211 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" Oct 30 16:59:11.779236 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.779200 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf"] Oct 30 16:59:11.783546 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.783522 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:11.804721 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.804293 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-04248e-serving-cert\"" Oct 30 16:59:11.826887 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.826845 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf"] Oct 30 16:59:11.893709 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.893674 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:11.893981 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.893718 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d34d00-c34b-4a89-bc00-06d8972e0ab0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:11.893981 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.893881 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-metrics-certs\") pod \"maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:11.893981 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.893959 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2md9\" (UniqueName: \"kubernetes.io/projected/14d34d00-c34b-4a89-bc00-06d8972e0ab0-kube-api-access-m2md9\") pod \"maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:11.897857 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.897821 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:11.994601 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.994562 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-metrics-certs\") pod \"maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:11.994601 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.994601 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2md9\" (UniqueName: \"kubernetes.io/projected/14d34d00-c34b-4a89-bc00-06d8972e0ab0-kube-api-access-m2md9\") pod \"maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:11.994855 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.994665 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:11.994855 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.994688 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d34d00-c34b-4a89-bc00-06d8972e0ab0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:11.994855 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:59:11.994797 4828 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-serving-cert: secret "maas-api-e2e-worker-w1-04248e-serving-cert" not found Oct 30 16:59:11.995005 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:59:11.994877 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-maas-api-tls podName:14d34d00-c34b-4a89-bc00-06d8972e0ab0 nodeName:}" failed. No retries permitted until 2026-10-30 16:59:12.494857057 +0000 UTC m=+730.140367230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-maas-api-tls") pod "maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" (UID: "14d34d00-c34b-4a89-bc00-06d8972e0ab0") : secret "maas-api-e2e-worker-w1-04248e-serving-cert" not found Oct 30 16:59:11.995326 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.995308 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d34d00-c34b-4a89-bc00-06d8972e0ab0-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:11.997298 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:11.997272 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-metrics-certs\") pod \"maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:12.020467 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:12.020432 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2md9\" (UniqueName: \"kubernetes.io/projected/14d34d00-c34b-4a89-bc00-06d8972e0ab0-kube-api-access-m2md9\") pod \"maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:12.048669 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:12.048577 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:12.222422 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:12.222394 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" Oct 30 16:59:12.231169 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:12.231132 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:12.286994 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:12.286960 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 16:59:12.500585 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:12.500505 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:12.503627 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:12.503589 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-maas-api-tls\") pod \"maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:12.702837 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:12.702803 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:13.021192 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:13.021148 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf"] Oct 30 16:59:13.391664 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:13.376212 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" event={"ID":"14d34d00-c34b-4a89-bc00-06d8972e0ab0","Type":"ContainerStarted","Data":"fab725fa1de936e4184f6f841035ffe43522f6eea6baf1d31c573abfed9e2a1f"} Oct 30 16:59:13.642312 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:13.642212 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:14.241554 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:14.241526 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" Oct 30 16:59:14.384313 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:14.384272 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" event={"ID":"14d34d00-c34b-4a89-bc00-06d8972e0ab0","Type":"ContainerStarted","Data":"f4f04bdf93064626e7d68814f0e8f412ab9be8c10aef681440f6e19f42fee773"} Oct 30 16:59:14.384509 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:14.384480 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:14.385846 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:14.385823 4828 generic.go:356] "Generic (PLEG): container finished" podID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerID="07db4d7706f33854aa2205387dbb67ba093f9bcc7f3e3a624319a67c9d0471ec" exitCode=0 Oct 30 16:59:14.385956 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:14.385889 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" event={"ID":"e2a75163-e6e5-4eaa-9197-a0e16de86f19","Type":"ContainerDied","Data":"07db4d7706f33854aa2205387dbb67ba093f9bcc7f3e3a624319a67c9d0471ec"} Oct 30 16:59:14.386264 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:14.386250 4828 scope.go:122] "RemoveContainer" containerID="07db4d7706f33854aa2205387dbb67ba093f9bcc7f3e3a624319a67c9d0471ec" Oct 30 16:59:14.387435 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:14.387406 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:14.387525 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:14.387489 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:14.387586 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:14.387537 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:14.431869 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:14.431817 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" podStartSLOduration=3.129874385 podStartE2EDuration="3.431800317s" totalImagesPullingTime="301.925932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:14.43119301 +0000 UTC m=+732.076703192" watchObservedRunningTime="2026-10-30 16:59:14.431800317 +0000 UTC m=+732.077310498" Oct 30 16:59:15.249304 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:15.249266 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" Oct 30 16:59:15.348518 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:15.348486 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 16:59:15.350825 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:15.350781 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:15.419255 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:15.419215 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" event={"ID":"e2a75163-e6e5-4eaa-9197-a0e16de86f19","Type":"ContainerStarted","Data":"0d85d09906cce7f74b3d058b6aaff289414b47a7ae9d2dda54f831f410203300"} Oct 30 16:59:15.514678 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:15.506320 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:15.658970 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:15.658929 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:15.664762 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:15.664649 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:59:15.664870 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:15.664804 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 16:59:15.664870 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:15.664843 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 16:59:16.077923 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:16.076471 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:16.238770 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:16.238598 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:16.665182 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:16.665130 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 16:59:16.666857 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:16.666818 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 16:59:17.437234 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:17.437157 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:17.664805 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:17.664773 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 16:59:17.665069 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:17.665038 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 16:59:18.664187 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:18.664154 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 16:59:18.664591 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:18.664205 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 16:59:19.493460 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:19.493422 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:19.664545 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:19.664496 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 16:59:19.664951 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:19.664559 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 16:59:20.149308 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:20.149271 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:20.448172 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:20.448088 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 16:59:20.666040 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:20.665882 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Oct 30 16:59:20.666654 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:20.666602 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Oct 30 16:59:20.844318 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:20.843637 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:21.008604 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:21.008568 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:21.185779 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:21.185689 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:21.801936 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:21.800314 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:21.836923 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:21.835615 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:59:21.843286 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:21.843192 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:59:21.881229 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:21.881194 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 16:59:21.895697 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:21.893208 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g"] Oct 30 16:59:21.927645 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:21.924890 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g"] Oct 30 16:59:22.044926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:21.965800 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.044926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.004718 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-d384dc-openshift-default-dockercfg-r4qzb\"" Oct 30 16:59:22.044926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.005614 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.044926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.005679 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7beb4126-0039-4932-be23-21d475782c17-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.044926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.005768 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7beb4126-0039-4932-be23-21d475782c17-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.044926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.005802 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.044926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.005832 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.044926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.005937 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h82vc\" (UniqueName: \"kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-kube-api-access-h82vc\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.044926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.006027 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.044926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.006070 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.044926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.006124 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.050845 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.050721 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:22.136967 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.117788 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.137988 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.137956 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.138188 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.138173 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.138308 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.138293 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7beb4126-0039-4932-be23-21d475782c17-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.138452 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.138439 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7beb4126-0039-4932-be23-21d475782c17-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.138565 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.138553 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.138675 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.138660 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.138819 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.138803 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82vc\" (UniqueName: \"kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-kube-api-access-h82vc\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.138993 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.138979 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.139510 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.139487 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.139873 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.139854 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.140233 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.140212 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.144941 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.144879 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.145528 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.145481 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.145624 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.145582 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7beb4126-0039-4932-be23-21d475782c17-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.149378 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.149347 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7beb4126-0039-4932-be23-21d475782c17-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.242961 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.242846 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82vc\" (UniqueName: \"kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-kube-api-access-h82vc\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.243249 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.243219 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.315927 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.315410 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:22.366473 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.366426 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:22.726921 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.726779 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:22.804491 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.804450 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g"] Oct 30 16:59:22.830213 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.830166 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:22.830469 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.830446 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:22.830733 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.830692 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:22.866181 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:22.865471 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" event={"ID":"7beb4126-0039-4932-be23-21d475782c17","Type":"ContainerStarted","Data":"71e2ee12b2a7bb43876f0dc983374846be09a15449310aa69fe43ab9acb43543"} Oct 30 16:59:23.895926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:23.890292 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" event={"ID":"7beb4126-0039-4932-be23-21d475782c17","Type":"ContainerStarted","Data":"434e045a242926fe7bba5213580aec59c028e52bd2b1ae7430d8dccb5bd18ce7"} Oct 30 16:59:24.407926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:24.375171 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:24.407926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:24.375306 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:59:24.407926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:24.375351 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" podUID="7beb4126-0039-4932-be23-21d475782c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 16:59:24.847436 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:24.847395 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:25.173242 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:25.172025 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:59:25.173242 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:25.172097 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:59:25.371944 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:25.371878 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:59:25.372280 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:25.372242 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" podUID="7beb4126-0039-4932-be23-21d475782c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 16:59:26.295998 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:26.295871 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:26.367778 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:26.367449 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:59:26.367778 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:26.367504 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" podUID="7beb4126-0039-4932-be23-21d475782c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 16:59:26.916857 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:26.916812 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:27.091440 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:27.090384 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:27.254767 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:27.254605 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:27.367618 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:27.367534 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:59:27.367618 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:27.367585 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" podUID="7beb4126-0039-4932-be23-21d475782c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 16:59:28.368345 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:28.367868 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:59:28.368345 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:28.367946 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" podUID="7beb4126-0039-4932-be23-21d475782c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 16:59:29.364655 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:29.364560 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:29.371290 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:29.369387 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:59:29.372303 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:29.371830 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" podUID="7beb4126-0039-4932-be23-21d475782c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 16:59:30.371542 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:30.371508 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:30.372065 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:30.371939 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:30.373003 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:30.372980 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 16:59:30.456365 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:30.456309 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" podStartSLOduration=9.456290395 podStartE2EDuration="9.456290395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:23.97188254 +0000 UTC m=+741.617392722" watchObservedRunningTime="2026-10-30 16:59:30.456290395 +0000 UTC m=+748.101800574" Oct 30 16:59:31.422978 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:31.422884 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:31.814665 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:31.580015 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:32.039396 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.039363 4828 generic.go:356] "Generic (PLEG): container finished" podID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerID="7e7972e3031c2ce34e5ec3e5b93f502d89aeb3035d4a4844d0123ba157569972" exitCode=0 Oct 30 16:59:32.039615 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.039441 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" event={"ID":"05de1e06-0e9b-46a5-9020-5d791da7654a","Type":"ContainerDied","Data":"7e7972e3031c2ce34e5ec3e5b93f502d89aeb3035d4a4844d0123ba157569972"} Oct 30 16:59:32.040378 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.039933 4828 scope.go:122] "RemoveContainer" containerID="7e7972e3031c2ce34e5ec3e5b93f502d89aeb3035d4a4844d0123ba157569972" Oct 30 16:59:32.041410 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.041381 4828 generic.go:356] "Generic (PLEG): container finished" podID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerID="a1381e78acbc22a2e8a2803e9bdc59c4ff74442edff9893a30ef23fbf2dc00e1" exitCode=0 Oct 30 16:59:32.041563 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.041450 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" event={"ID":"36000b4a-d11f-483e-9a73-e5bdf70daa5a","Type":"ContainerDied","Data":"a1381e78acbc22a2e8a2803e9bdc59c4ff74442edff9893a30ef23fbf2dc00e1"} Oct 30 16:59:32.041646 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.041568 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:32.041710 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.041652 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:32.041878 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.041705 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:32.041981 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.041883 4828 scope.go:122] "RemoveContainer" containerID="a1381e78acbc22a2e8a2803e9bdc59c4ff74442edff9893a30ef23fbf2dc00e1" Oct 30 16:59:32.045742 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.045705 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:32.045873 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.045780 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:32.045873 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.045820 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:32.046632 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.046253 4828 generic.go:356] "Generic (PLEG): container finished" podID="9edde938-9f51-43c7-9411-128b417a3edc" containerID="ed356183a53424b1347f9f31609e026763d711c1eb9fc71083ccac7f09c22347" exitCode=0 Oct 30 16:59:32.046632 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.046354 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" event={"ID":"9edde938-9f51-43c7-9411-128b417a3edc","Type":"ContainerDied","Data":"ed356183a53424b1347f9f31609e026763d711c1eb9fc71083ccac7f09c22347"} Oct 30 16:59:32.047299 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.046941 4828 scope.go:122] "RemoveContainer" containerID="ed356183a53424b1347f9f31609e026763d711c1eb9fc71083ccac7f09c22347" Oct 30 16:59:32.048460 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.048311 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:32.048460 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.048386 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:32.048460 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.048429 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:32.111949 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.109972 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:32.338523 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.338430 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:32.494776 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:32.494739 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:33.130927 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:33.124162 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" event={"ID":"9edde938-9f51-43c7-9411-128b417a3edc","Type":"ContainerStarted","Data":"704c6269040e68ca3140a3c64e5cb2b7b04ddb2fad49d1be0643c28726fa36fc"} Oct 30 16:59:33.138923 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:33.136544 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" event={"ID":"05de1e06-0e9b-46a5-9020-5d791da7654a","Type":"ContainerStarted","Data":"a04e662daadb2596a48ca94b7233e465579e84f5adf83e326bbce3612e95305b"} Oct 30 16:59:33.153925 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:33.153433 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" event={"ID":"36000b4a-d11f-483e-9a73-e5bdf70daa5a","Type":"ContainerStarted","Data":"ba65c7e67874b7b51f098b04e588244995781d75c0a054b8af6c4c4ef9c4ddc4"} Oct 30 16:59:33.159710 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:33.159264 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:59:33.159710 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:33.159378 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:59:33.159710 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:33.159421 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 16:59:33.528352 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:33.528315 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:59:33.529056 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:33.529031 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:59:33.529228 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:33.529202 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 16:59:33.641923 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:33.640210 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:33.824379 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:33.824292 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:59:33.824709 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:33.824614 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 16:59:33.824868 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:33.824846 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 16:59:34.160206 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:34.159563 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:59:34.160206 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:34.159638 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 16:59:34.529414 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:34.529374 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:59:34.530046 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:34.530014 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 16:59:34.824984 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:34.824356 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 16:59:34.824984 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:34.824424 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 16:59:35.155970 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:35.155877 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:59:35.155970 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:35.155943 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 16:59:35.526925 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:35.522341 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:59:35.526925 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:35.522402 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 16:59:35.822983 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:35.822322 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 16:59:35.822983 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:35.822386 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 16:59:36.096974 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:36.090044 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:36.161935 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:36.160320 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:59:36.161935 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:36.160388 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 16:59:36.549926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:36.544215 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:59:36.549926 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:36.544276 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 16:59:36.646965 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:36.634285 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:36.812819 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:36.812718 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:36.823396 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:36.823364 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 16:59:36.824100 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:36.824068 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 16:59:37.156980 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:37.156381 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:59:37.156980 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:37.156453 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 16:59:37.202947 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:37.195670 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:37.353306 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:37.353270 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:37.521947 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:37.516723 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:37.522623 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:37.522596 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:59:37.522854 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:37.522813 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 16:59:37.821403 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:37.821308 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 16:59:37.821682 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:37.821625 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 16:59:38.035948 ip-10-0-135-153 kubenswrapper[4828]: E1030 16:59:38.035886 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 16:59:38.158742 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:38.158042 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:59:38.158742 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:38.158118 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 30 16:59:38.522935 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:38.521738 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:59:38.522935 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:38.521797 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 16:59:38.644534 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:38.644491 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:38.821738 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:38.821634 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Oct 30 16:59:38.821738 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:38.821695 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Oct 30 16:59:39.105447 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:39.105351 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:59:39.105953 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:39.105484 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:59:39.105953 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:39.105550 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:59:39.161347 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:39.161314 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:59:39.286958 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:39.286929 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 16:59:39.526576 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:39.526539 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:59:39.527711 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:39.527693 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 16:59:39.826083 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:39.825996 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:59:39.827090 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:39.827067 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 16:59:40.445001 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:40.444966 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:42.094731 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:42.094627 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:43.751052 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:43.751016 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:43.912751 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:43.912687 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:45.464703 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:45.464665 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:45.624698 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:45.624665 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:47.287321 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:47.287289 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:48.909881 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:48.906640 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:50.708965 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:50.708923 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:51.941124 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:51.941077 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:52.148959 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.148928 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc"] Oct 30 16:59:52.152878 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.152854 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.189459 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.189427 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.189616 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.189468 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.189616 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.189490 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.189695 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.189618 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.189695 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.189658 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.189763 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.189723 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.189796 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.189761 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.189796 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.189777 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748eedeb-7a6c-4bc9-b141-29be1431024a-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.189884 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.189868 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55vt\" (UniqueName: \"kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-kube-api-access-f55vt\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.210757 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.210678 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc"] Oct 30 16:59:52.290876 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.290839 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.291076 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.290890 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.291076 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.290947 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.291172 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.291128 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.291236 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.291194 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748eedeb-7a6c-4bc9-b141-29be1431024a-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.291313 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.291290 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55vt\" (UniqueName: \"kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-kube-api-access-f55vt\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.291446 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.291328 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.291511 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.291440 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.291511 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.291486 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.291631 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.291521 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.291684 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.291654 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.291979 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.291955 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.292132 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.292108 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.292253 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.292233 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748eedeb-7a6c-4bc9-b141-29be1431024a-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.293834 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.293812 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.293943 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.293812 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.321104 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.321070 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55vt\" (UniqueName: \"kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-kube-api-access-f55vt\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.331614 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.331588 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.466790 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.466697 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:52.765354 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.765311 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:52.802880 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.802832 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc"] Oct 30 16:59:52.806159 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.806123 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:52.806245 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.806198 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:52.806245 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:52.806227 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 16:59:53.252171 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:53.252075 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:53.402524 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:53.402484 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" event={"ID":"748eedeb-7a6c-4bc9-b141-29be1431024a","Type":"ContainerStarted","Data":"78385a15ac2e4f1e16197a7ae49f17ff304512a0696e4f5fbab522505417732a"} Oct 30 16:59:53.402717 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:53.402555 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" event={"ID":"748eedeb-7a6c-4bc9-b141-29be1431024a","Type":"ContainerStarted","Data":"05727e134a594f5df0a7340bd899dd27bdaaf3dc5cb02520896e2725dcd72792"} Oct 30 16:59:53.467391 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:53.467358 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:53.480225 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:53.480181 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:59:53.480386 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:53.480323 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" podUID="748eedeb-7a6c-4bc9-b141-29be1431024a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 16:59:53.919221 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:53.919183 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:54.467281 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:54.467248 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:59:54.467676 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:54.467303 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" podUID="748eedeb-7a6c-4bc9-b141-29be1431024a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 16:59:54.909924 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:54.908825 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:55.467623 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:55.467585 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:59:55.468077 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:55.468052 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" podUID="748eedeb-7a6c-4bc9-b141-29be1431024a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 16:59:55.556111 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:55.556080 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:56.468002 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:56.467970 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:59:56.468416 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:56.468022 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" podUID="748eedeb-7a6c-4bc9-b141-29be1431024a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 16:59:56.582942 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:56.579940 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:57.467725 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:57.467694 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:59:57.467942 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:57.467745 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" podUID="748eedeb-7a6c-4bc9-b141-29be1431024a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 16:59:58.477523 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:58.477480 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:59:58.489053 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:58.489013 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" podUID="748eedeb-7a6c-4bc9-b141-29be1431024a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 16:59:58.489646 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:58.486230 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 16:59:59.471736 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:59.471701 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:59.472095 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:59.472073 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:59.473005 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:59.472983 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 16:59:59.569453 ip-10-0-135-153 kubenswrapper[4828]: I1030 16:59:59.569393 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" podStartSLOduration=7.5693802550000004 podStartE2EDuration="7.569380255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:53.464284555 +0000 UTC m=+771.109794735" watchObservedRunningTime="2026-10-30 16:59:59.569380255 +0000 UTC m=+777.214890435" Oct 30 17:00:00.444160 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:00.444120 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:02.300030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:02.299572 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:03.978773 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:03.978733 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:05.673298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:05.673207 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:07.332595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:07.330189 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:07.914340 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:00:07.914298 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:00:08.099370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:08.099337 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:08.979497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:08.979442 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:10.655477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:10.655427 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:12.355763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:12.355726 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:13.894677 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:13.894638 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:15.435727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:15.435684 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:17.269594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:17.269550 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:18.934884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:18.934840 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:19.168580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:19.168544 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:20.471081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:20.471042 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:22.270973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:22.267861 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:24.044923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:24.044611 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:25.109183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.109143 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m"] Oct 30 17:00:25.113922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.113886 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.133484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.133410 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m"] Oct 30 17:00:25.142764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.142733 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b4f406\"/\"default-dockercfg-gf4d8\"" Oct 30 17:00:25.142943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.142821 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b4f406\"/\"kube-root-ca.crt\"" Oct 30 17:00:25.142943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.142821 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b4f406\"/\"openshift-service-ca.crt\"" Oct 30 17:00:25.142943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.142831 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b4f406\"/\"premium-w2-b4f406-kserve-self-signed-certs\"" Oct 30 17:00:25.175371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.175322 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg495\" (UniqueName: \"kubernetes.io/projected/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-kube-api-access-bg495\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.175594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.175376 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tls-certs\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.175594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.175454 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-dshm\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.175594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.175524 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tmp-dir\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.175783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.175606 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-home\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.175783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.175657 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-model-cache\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.214142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.214104 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:25.276504 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.276463 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-dshm\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.276697 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.276542 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tmp-dir\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.276697 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.276618 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-home\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.276697 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.276673 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-model-cache\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.276868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.276713 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg495\" (UniqueName: \"kubernetes.io/projected/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-kube-api-access-bg495\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.277360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.277056 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-home\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.277360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.277088 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tmp-dir\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.277360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.277124 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tls-certs\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.277360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.277130 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-model-cache\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.278974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.278946 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-dshm\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.279439 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.279421 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tls-certs\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.328768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.328727 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg495\" (UniqueName: \"kubernetes.io/projected/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-kube-api-access-bg495\") pod \"premium-w2-b4f406-kserve-77d46c6c94-qc99m\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.428208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.428127 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:25.697335 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.694934 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:25.779070 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:00:25.779037 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabc0ea64_db5b_4cf9_abcf_cb18b4fadf1f.slice/crio-ebcad0ecb256990a0192db2a50899ebb9ca3fa7c5a6d36bcf58dcd8c3da0f6b8 WatchSource:0}: Error finding container ebcad0ecb256990a0192db2a50899ebb9ca3fa7c5a6d36bcf58dcd8c3da0f6b8: Status 404 returned error can't find the container with id ebcad0ecb256990a0192db2a50899ebb9ca3fa7c5a6d36bcf58dcd8c3da0f6b8 Oct 30 17:00:25.780171 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:25.780141 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m"] Oct 30 17:00:26.752263 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:26.752219 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" event={"ID":"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f","Type":"ContainerStarted","Data":"aa4696d145d000c278f6b32801c26fc71baa181a26a9070bdd1cb138be78c2f4"} Oct 30 17:00:26.752744 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:26.752271 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" event={"ID":"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f","Type":"ContainerStarted","Data":"ebcad0ecb256990a0192db2a50899ebb9ca3fa7c5a6d36bcf58dcd8c3da0f6b8"} Oct 30 17:00:26.801254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:26.801183 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podStartSLOduration=1.801168426 podStartE2EDuration="1.801168426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:26.800641711 +0000 UTC m=+804.446151898" watchObservedRunningTime="2026-10-30 17:00:26.801168426 +0000 UTC m=+804.446678606" Oct 30 17:00:27.428608 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:27.428569 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:27.491933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:27.491876 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:28.619752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:28.619711 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:29.167295 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:29.167254 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:30.959911 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:30.959859 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:32.527081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:32.527030 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:34.158915 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:34.158856 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:34.325447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:34.325409 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:35.432922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:35.432518 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:35.534503 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:35.534462 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:35.888945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:35.888887 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:00:36.180549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:36.180456 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:37.319844 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.319798 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d"] Oct 30 17:00:37.325847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.325819 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.347260 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.347233 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-86f50a-openshift-default-dockercfg-n29ph\"" Oct 30 17:00:37.361527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.361497 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d"] Oct 30 17:00:37.487679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.487640 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:37.503832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.503796 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.504017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.503840 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.504083 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.504058 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.504133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.504115 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrvnc\" (UniqueName: \"kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-kube-api-access-xrvnc\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.504176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.504155 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.504214 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.504194 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.504256 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.504216 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.504256 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.504245 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.504365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.504347 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa9acde-75ca-4422-ae94-ab6dbd630313-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.606097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.606017 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.606097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.606063 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.606531 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.606506 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.606660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.606550 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.606660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.606598 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrvnc\" (UniqueName: \"kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-kube-api-access-xrvnc\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.606660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.606646 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.606835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.606717 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.606835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.606757 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.606835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.606801 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.606835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.606815 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.607067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.606840 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa9acde-75ca-4422-ae94-ab6dbd630313-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.607350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.607323 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.607569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.607547 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.607684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.607613 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa9acde-75ca-4422-ae94-ab6dbd630313-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.608887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.608861 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.609050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.609033 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.633548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.633137 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.636647 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.636581 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrvnc\" (UniqueName: \"kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-kube-api-access-xrvnc\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.647336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:37.647139 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:37.912161 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:00:37.912071 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:00:38.049561 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:00:38.049530 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa9acde_75ca_4422_ae94_ab6dbd630313.slice/crio-1ed9997b3fa69d28bf0260976b01838cd785d39ddb5dc9b1e30ccee8d57af295 WatchSource:0}: Error finding container 1ed9997b3fa69d28bf0260976b01838cd785d39ddb5dc9b1e30ccee8d57af295: Status 404 returned error can't find the container with id 1ed9997b3fa69d28bf0260976b01838cd785d39ddb5dc9b1e30ccee8d57af295 Oct 30 17:00:38.049726 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:38.049704 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d"] Oct 30 17:00:38.051813 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:38.051782 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:38.051875 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:38.051848 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:38.051961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:38.051876 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:38.189139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:38.186368 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:38.910498 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:38.910460 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" event={"ID":"5aa9acde-75ca-4422-ae94-ab6dbd630313","Type":"ContainerStarted","Data":"64f0496dd22662f2d854bf8970100ba8b29f9f392bb15dda6de1245a0e4e2252"} Oct 30 17:00:38.910987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:38.910530 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" event={"ID":"5aa9acde-75ca-4422-ae94-ab6dbd630313","Type":"ContainerStarted","Data":"1ed9997b3fa69d28bf0260976b01838cd785d39ddb5dc9b1e30ccee8d57af295"} Oct 30 17:00:39.000290 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:39.000227 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" podStartSLOduration=2.000206459 podStartE2EDuration="2.000206459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:38.982425734 +0000 UTC m=+816.627935915" watchObservedRunningTime="2026-10-30 17:00:39.000206459 +0000 UTC m=+816.645716641" Oct 30 17:00:39.667120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:39.667082 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:39.669085 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:39.669053 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:00:39.669361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:39.669331 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" podUID="5aa9acde-75ca-4422-ae94-ab6dbd630313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 17:00:39.975974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:39.975862 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:40.667979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:40.667944 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:00:40.668157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:40.668001 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" podUID="5aa9acde-75ca-4422-ae94-ab6dbd630313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 17:00:41.499341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:41.499310 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:41.667163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:41.667128 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:00:41.667377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:41.667187 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" podUID="5aa9acde-75ca-4422-ae94-ab6dbd630313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 17:00:41.668957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:41.668842 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:42.667573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:42.667540 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:00:42.668012 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:42.667594 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" podUID="5aa9acde-75ca-4422-ae94-ab6dbd630313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 17:00:43.667667 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:43.667627 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 17:00:43.668208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:43.667686 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" podUID="5aa9acde-75ca-4422-ae94-ab6dbd630313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 17:00:43.792891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:43.792795 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:44.957421 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:44.957390 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:44.958693 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:44.958673 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:44.959710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:44.959687 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:00:45.447916 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:45.447868 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:47.420553 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:47.420516 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:49.484056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:49.484015 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:51.462103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:51.462059 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:52.084159 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.084123 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw"] Oct 30 17:00:52.088307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.088280 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.120849 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.120811 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw"] Oct 30 17:00:52.243440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.243407 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.243629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.243535 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.243629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.243583 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.243629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.243607 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.243629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.243627 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.243799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.243778 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.243850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.243825 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfp87\" (UniqueName: \"kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-kube-api-access-hfp87\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.243915 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.243849 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.243966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.243891 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.345709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.345607 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.345709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.345666 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.345709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.345687 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.346041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.345722 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.346041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.345773 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.346041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.345817 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfp87\" (UniqueName: \"kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-kube-api-access-hfp87\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.346041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.345844 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.346041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.345867 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.346041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.345955 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.346345 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.346209 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.346345 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.346236 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.346613 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.346565 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.346613 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.346587 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.346794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.346594 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.348521 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.348498 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.348706 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.348562 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.360075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.360041 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6"] Oct 30 17:00:52.364410 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.364378 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.395449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.395414 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfp87\" (UniqueName: \"kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-kube-api-access-hfp87\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.402964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.402928 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.404781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.404758 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:52.423793 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.423762 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6"] Oct 30 17:00:52.453099 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.453061 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.453292 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.453129 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.453292 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.453190 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08423875-76fe-41f6-995a-6c5832230d12-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.453292 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.453230 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4xg\" (UniqueName: \"kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-kube-api-access-mz4xg\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.453541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.453258 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.453541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.453350 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.453541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.453446 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.453783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.453495 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08423875-76fe-41f6-995a-6c5832230d12-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.453783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.453616 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.556665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.556619 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.561231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.556705 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.561231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.556759 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08423875-76fe-41f6-995a-6c5832230d12-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.561231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.556807 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4xg\" (UniqueName: \"kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-kube-api-access-mz4xg\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.561231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.556843 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.561231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.556867 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.561231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.556973 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.561231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.557048 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08423875-76fe-41f6-995a-6c5832230d12-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.561231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.557109 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.561231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.557831 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.561231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.558463 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.561231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.558834 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08423875-76fe-41f6-995a-6c5832230d12-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.561231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.559279 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.561231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.560430 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.564501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.564479 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08423875-76fe-41f6-995a-6c5832230d12-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.566073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.566050 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.591052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.590981 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.595324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.595264 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4xg\" (UniqueName: \"kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-kube-api-access-mz4xg\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.685708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.685609 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:52.752213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.752146 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw"] Oct 30 17:00:52.755470 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:00:52.755439 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff93c7e0_92d5_4d7a_ba3b_ab36a0ef4167.slice/crio-e83a4372c5d16aa37e7d7ba40878f50afb28eb27f392d4080b030c668912ef68 WatchSource:0}: Error finding container e83a4372c5d16aa37e7d7ba40878f50afb28eb27f392d4080b030c668912ef68: Status 404 returned error can't find the container with id e83a4372c5d16aa37e7d7ba40878f50afb28eb27f392d4080b030c668912ef68 Oct 30 17:00:52.758183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.758146 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:52.758297 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.758257 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:52.758362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:52.758305 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:53.005707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:53.005679 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6"] Oct 30 17:00:53.008064 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:53.008023 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:53.008148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:53.008116 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:53.008196 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:53.008161 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:00:53.032442 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:53.032407 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" event={"ID":"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167","Type":"ContainerStarted","Data":"e0f7bc0e13a76fb6e65db58ce02b70b07cf030b7771946f2c57c1f758a40f260"} Oct 30 17:00:53.032585 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:53.032471 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" event={"ID":"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167","Type":"ContainerStarted","Data":"e83a4372c5d16aa37e7d7ba40878f50afb28eb27f392d4080b030c668912ef68"} Oct 30 17:00:53.033874 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:53.033846 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" event={"ID":"08423875-76fe-41f6-995a-6c5832230d12","Type":"ContainerStarted","Data":"ecfd8fe29f8f3f4af89d0ada96b0b9a1595c1bc69cb55b4d80b0ab723fcd4ccc"} Oct 30 17:00:53.099268 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:53.099197 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" podStartSLOduration=1.099178035 podStartE2EDuration="1.099178035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:53.093310234 +0000 UTC m=+830.738820428" watchObservedRunningTime="2026-10-30 17:00:53.099178035 +0000 UTC m=+830.744688215" Oct 30 17:00:53.405265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:53.405231 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:00:53.407470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:53.407436 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:00:53.407628 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:53.407500 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" podUID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 17:00:53.868926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:53.864419 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:54.046643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:54.046566 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" event={"ID":"08423875-76fe-41f6-995a-6c5832230d12","Type":"ContainerStarted","Data":"f352ae92bc51d628d40c4a30337cef2054019f29066e81152bcbb51b6b855f53"} Oct 30 17:00:54.124203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:54.124085 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" podStartSLOduration=2.124064771 podStartE2EDuration="2.124064771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:54.107471025 +0000 UTC m=+831.752981201" watchObservedRunningTime="2026-10-30 17:00:54.124064771 +0000 UTC m=+831.769574951" Oct 30 17:00:54.132861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:54.132833 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:54.300923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:54.293730 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:54.418975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:54.418354 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:00:54.418975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:54.418417 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" podUID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 17:00:54.703177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:54.703081 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:00:54.708053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:54.708019 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 17:00:54.708307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:54.708280 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" podUID="08423875-76fe-41f6-995a-6c5832230d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 17:00:55.405321 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:55.405279 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:00:55.405972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:55.405937 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" podUID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 17:00:55.689802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:55.689282 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 17:00:55.689802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:55.689344 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" podUID="08423875-76fe-41f6-995a-6c5832230d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 17:00:56.164941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:56.163184 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:56.405924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:56.405300 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:00:56.405924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:56.405362 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" podUID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 17:00:56.690348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:56.690306 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 17:00:56.690642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:56.690608 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" podUID="08423875-76fe-41f6-995a-6c5832230d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 17:00:57.407356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:57.407319 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:00:57.407964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:57.407928 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" podUID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 17:00:57.690336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:57.690239 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 17:00:57.690336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:57.690303 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" podUID="08423875-76fe-41f6-995a-6c5832230d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 17:00:58.406924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:58.406268 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:00:58.406924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:58.406335 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" podUID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 17:00:58.686981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:58.686030 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 17:00:58.686981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:58.686092 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" podUID="08423875-76fe-41f6-995a-6c5832230d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 17:00:58.763695 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:58.763163 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:00:59.405558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:59.405513 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 17:00:59.405784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:59.405590 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" podUID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 17:00:59.686991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:59.686866 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 17:00:59.686991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:00:59.686945 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" podUID="08423875-76fe-41f6-995a-6c5832230d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 17:01:00.409753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:00.409721 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:01:00.410131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:00.410110 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:01:00.410704 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:00.410688 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:01:00.541316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:00.541275 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:00.691129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:00.691050 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:01:00.691518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:00.691497 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:01:00.692238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:00.692220 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:01:02.409922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:02.398538 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:04.218153 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:04.218119 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:06.113681 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:06.113541 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:07.752635 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.752597 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt"] Oct 30 17:01:07.757855 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.757828 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.779035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.778998 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-096d75-openshift-default-dockercfg-twcrb\"" Oct 30 17:01:07.793002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.792965 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt"] Oct 30 17:01:07.829315 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.829267 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.829489 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.829326 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.829489 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.829412 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.829578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.829504 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.829578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.829566 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.829689 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.829593 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7rv5\" (UniqueName: \"kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-kube-api-access-p7rv5\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.829793 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.829771 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.829858 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.829822 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.829965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.829929 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.911597 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:01:07.911557 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:01:07.930818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.930777 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.931698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.930861 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.931698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.930950 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.931698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.930971 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.931698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.930999 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.931698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.931050 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.931698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.931106 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.931698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.931127 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7rv5\" (UniqueName: \"kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-kube-api-access-p7rv5\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.931698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.931170 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.931698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.931569 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.931698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.931625 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.931698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.931641 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.932243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.931781 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.932243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.931819 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.933461 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.933432 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.933797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.933779 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.956318 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.956289 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:07.956473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:07.956455 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7rv5\" (UniqueName: \"kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-kube-api-access-p7rv5\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:08.072513 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:08.072477 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:08.408276 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:08.408245 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt"] Oct 30 17:01:08.409445 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:01:08.409421 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7a2ea3_7b3e_4cbe_a78a_da2073164c6d.slice/crio-e5705ae3763a8a670498eee6d6f28b0fc50b4b22c4d3c2c8dd335a7890b45d35 WatchSource:0}: Error finding container e5705ae3763a8a670498eee6d6f28b0fc50b4b22c4d3c2c8dd335a7890b45d35: Status 404 returned error can't find the container with id e5705ae3763a8a670498eee6d6f28b0fc50b4b22c4d3c2c8dd335a7890b45d35 Oct 30 17:01:08.412482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:08.412449 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:08.412570 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:08.412514 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:08.412570 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:08.412543 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:08.512942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:08.510371 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" event={"ID":"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d","Type":"ContainerStarted","Data":"ec827807ba76bfab23adc5a6fc8ba8c146b5e59467bbd341f963a894e3a24c23"} Oct 30 17:01:08.512942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:08.510433 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" event={"ID":"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d","Type":"ContainerStarted","Data":"e5705ae3763a8a670498eee6d6f28b0fc50b4b22c4d3c2c8dd335a7890b45d35"} Oct 30 17:01:08.561156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:08.561091 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" podStartSLOduration=1.5610703240000001 podStartE2EDuration="1.561070324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:08.560703747 +0000 UTC m=+846.206213931" watchObservedRunningTime="2026-10-30 17:01:08.561070324 +0000 UTC m=+846.206580507" Oct 30 17:01:09.073091 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:09.073052 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:09.075185 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:09.075158 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:01:09.075300 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:09.075209 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:01:09.259833 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:09.239572 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:10.076925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:10.075284 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:01:10.076925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:10.075345 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:01:11.077924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:11.073370 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:01:11.077924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:11.073448 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:01:11.493974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:11.491888 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:12.076925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:12.073311 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:01:12.076925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:12.073377 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:01:13.077926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:13.076924 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:01:13.077926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:13.076990 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:01:13.380975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:13.380568 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:14.073924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:14.073309 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:01:14.073924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:14.073374 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:01:15.073402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:15.073361 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 17:01:15.073797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:15.073434 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 17:01:15.220989 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:15.218013 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:16.077966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:16.077934 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:16.078440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:16.078394 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:16.079333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:16.079314 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:01:17.030619 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:17.030578 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:18.720478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:18.720444 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:20.336756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:20.336709 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:22.834984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:22.834945 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:24.669070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:24.669036 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:26.752166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:26.752107 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:28.385721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:28.385670 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:30.092423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:30.092384 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:31.593763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:31.593726 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:32.142305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:32.142259 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:33.773034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:33.772934 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:35.629935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:35.629877 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:37.894065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:37.892920 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g"] Oct 30 17:01:37.930930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:37.918453 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:37.944934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:37.936885 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g"] Oct 30 17:01:38.103084 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.103038 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.139086 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.139043 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.145213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.145116 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.181109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.181064 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.181392 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.181365 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.181576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.181557 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.181711 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.181694 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab33acc-67cf-4cc4-b180-50cdee10b872-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.181893 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.181873 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhfwk\" (UniqueName: \"kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-kube-api-access-hhfwk\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.182120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.182102 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.294670 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.294619 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.294973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.294944 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.295172 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.295155 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.295325 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.295311 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.295481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.295467 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.295590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.295576 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.295725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.295712 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.295833 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.295818 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab33acc-67cf-4cc4-b180-50cdee10b872-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.296009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.295990 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhfwk\" (UniqueName: \"kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-kube-api-access-hhfwk\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.297170 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.296960 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.297584 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.297560 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.298520 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.298492 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.300381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.298877 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.300578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.300330 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab33acc-67cf-4cc4-b180-50cdee10b872-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.333990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.333956 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.347888 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.334957 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.372951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.372894 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.374551 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.374493 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhfwk\" (UniqueName: \"kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-kube-api-access-hhfwk\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.438968 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:01:38.438801 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:01:38.645709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.645677 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:38.902174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:38.902134 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:39.538786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:39.538670 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g"] Oct 30 17:01:39.547818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:39.547777 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:39.548109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:39.548078 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:39.548263 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:39.548245 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:40.013960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:40.013924 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" event={"ID":"aab33acc-67cf-4cc4-b180-50cdee10b872","Type":"ContainerStarted","Data":"f4ed8258ac28f87b1f355565bdd0e6b6e254f011779336a54bc7df94510834c2"} Oct 30 17:01:40.014518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:40.014486 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" event={"ID":"aab33acc-67cf-4cc4-b180-50cdee10b872","Type":"ContainerStarted","Data":"081061cca22f7451f23b3c725fa6849b150274aaf1e02bb7a818c68427444140"} Oct 30 17:01:40.077783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:40.077721 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" podStartSLOduration=3.077703896 podStartE2EDuration="3.077703896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:40.077686248 +0000 UTC m=+877.723196443" watchObservedRunningTime="2026-10-30 17:01:40.077703896 +0000 UTC m=+877.723214073" Oct 30 17:01:40.654683 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:40.654648 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:40.655983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:40.655952 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:01:40.656108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:40.656006 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" podUID="aab33acc-67cf-4cc4-b180-50cdee10b872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 17:01:40.964280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:40.957143 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:41.655601 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:41.655568 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:01:41.655994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:41.655629 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" podUID="aab33acc-67cf-4cc4-b180-50cdee10b872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 17:01:42.654721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:42.654680 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:01:42.654927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:42.654744 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" podUID="aab33acc-67cf-4cc4-b180-50cdee10b872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 17:01:42.821648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:42.821602 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:43.655067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:43.655031 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:01:43.655244 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:43.655085 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" podUID="aab33acc-67cf-4cc4-b180-50cdee10b872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 17:01:44.598921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:44.589788 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:44.655592 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:44.655554 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:01:44.655765 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:44.655607 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" podUID="aab33acc-67cf-4cc4-b180-50cdee10b872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 17:01:45.655643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:45.655607 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 17:01:45.656192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:45.655675 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" podUID="aab33acc-67cf-4cc4-b180-50cdee10b872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 17:01:46.453923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:46.446511 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:46.683197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:46.683164 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:46.683973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:46.683947 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:46.685535 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:46.685489 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:01:48.124926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.124524 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:48.842548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.842517 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl"] Oct 30 17:01:48.846668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.846648 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:48.875323 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.875285 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl"] Oct 30 17:01:48.882333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.882305 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn"] Oct 30 17:01:48.888884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.887638 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:48.910346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.910311 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:48.912331 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.912301 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:48.912582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.912561 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:48.912735 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.912717 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:48.912914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.912878 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g9bh\" (UniqueName: \"kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-kube-api-access-7g9bh\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:48.913177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.913158 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:48.913352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.913327 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:48.913560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.913474 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:48.913560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.913509 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:48.927573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:48.927537 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn"] Oct 30 17:01:49.014824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.014790 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.015070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.014844 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.015070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.014937 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.015070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015042 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.015234 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015080 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.015234 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015129 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.015234 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015155 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.015382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015294 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.015382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015333 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.015382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015358 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.015382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015377 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.015606 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015583 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.015687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015668 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.015740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015720 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.015740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015727 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g9bh\" (UniqueName: \"kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-kube-api-access-7g9bh\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.015973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015948 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.016066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.016012 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.016066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.016046 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc52h\" (UniqueName: \"kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-kube-api-access-wc52h\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.016066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.015959 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.016217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.016163 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.016217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.016186 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.016217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.016193 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.017125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.017100 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.020953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.018226 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.020953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.018258 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.040586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.040555 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.040775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.040751 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g9bh\" (UniqueName: \"kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-kube-api-access-7g9bh\") pod \"maas-default-gateway-openshift-default-6db586dd6c-dccbl\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.118398 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.118297 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.118586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.118461 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.118586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.118498 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.118586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.118522 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.118586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.118567 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.118796 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.118596 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.118796 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.118637 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.118796 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.118683 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.118998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.118798 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc52h\" (UniqueName: \"kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-kube-api-access-wc52h\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.119130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.119103 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.119205 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.119183 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.119289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.119265 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.119376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.119353 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.119491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.119381 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.120950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.120928 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.121449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.121431 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.143602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.143570 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.143965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.143656 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc52h\" (UniqueName: \"kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-kube-api-access-wc52h\") pod \"maas-default-gateway-openshift-default-6db586dd6c-kkhmn\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.162662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.162632 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:49.234993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.234962 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:49.489497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.489413 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl"] Oct 30 17:01:49.494432 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:01:49.494393 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469bb33d_3a10_4c4b_9ac0_e1f1262e8fda.slice/crio-d361246501d067135af941d0034a22ff5de3a2409e925f49381925b624740dbe WatchSource:0}: Error finding container d361246501d067135af941d0034a22ff5de3a2409e925f49381925b624740dbe: Status 404 returned error can't find the container with id d361246501d067135af941d0034a22ff5de3a2409e925f49381925b624740dbe Oct 30 17:01:49.497569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.497529 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:49.497708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.497606 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:49.497708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.497649 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:49.582589 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.582551 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn"] Oct 30 17:01:49.582767 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:01:49.582727 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7750e48_b71a_4a12_94b3_f5ab86d3d40c.slice/crio-ed00b9d929b27850104c3e47d6e7aef9c7d98b560acab980b7c7d6604bd8bd39 WatchSource:0}: Error finding container ed00b9d929b27850104c3e47d6e7aef9c7d98b560acab980b7c7d6604bd8bd39: Status 404 returned error can't find the container with id ed00b9d929b27850104c3e47d6e7aef9c7d98b560acab980b7c7d6604bd8bd39 Oct 30 17:01:49.585972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.585696 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:49.585972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.585781 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:49.585972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.585822 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:49.732938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:49.725423 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:50.114943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:50.114889 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" event={"ID":"c7750e48-b71a-4a12-94b3-f5ab86d3d40c","Type":"ContainerStarted","Data":"ec82117210e1cd6b23b620078bfb265ed343114aaf349d8d40d07a6e4876d810"} Oct 30 17:01:50.115118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:50.114968 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" event={"ID":"c7750e48-b71a-4a12-94b3-f5ab86d3d40c","Type":"ContainerStarted","Data":"ed00b9d929b27850104c3e47d6e7aef9c7d98b560acab980b7c7d6604bd8bd39"} Oct 30 17:01:50.117760 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:50.117726 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" event={"ID":"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda","Type":"ContainerStarted","Data":"c2dadf7940e2377c11d59e266cd28419726a7bf28c83b55801479ed9430ff466"} Oct 30 17:01:50.117875 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:50.117784 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" event={"ID":"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda","Type":"ContainerStarted","Data":"d361246501d067135af941d0034a22ff5de3a2409e925f49381925b624740dbe"} Oct 30 17:01:50.162806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:50.162771 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:01:50.164685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:50.164652 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:01:50.164826 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:50.164710 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:01:50.166456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:50.166414 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podStartSLOduration=2.166401638 podStartE2EDuration="2.166401638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:50.163206309 +0000 UTC m=+887.808716509" watchObservedRunningTime="2026-10-30 17:01:50.166401638 +0000 UTC m=+887.811911812" Oct 30 17:01:50.238282 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:50.238248 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:01:50.241657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:50.240019 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:01:50.241657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:50.240073 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:01:51.164019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:51.163981 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:01:51.164426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:51.164042 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:01:51.239335 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:51.239299 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:01:51.239587 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:51.239557 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:01:51.779925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:51.776313 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:51.966099 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:51.966065 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:52.159974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.153749 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podStartSLOduration=4.153723918 podStartE2EDuration="4.153723918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:50.234933734 +0000 UTC m=+887.880443915" watchObservedRunningTime="2026-10-30 17:01:52.153723918 +0000 UTC m=+889.799234099" Oct 30 17:01:52.218992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.171775 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:01:52.219682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.206799 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c"] Oct 30 17:01:52.331489 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.219547 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:01:52.331489 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.267006 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:01:52.331489 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.330470 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:01:52.467607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.467528 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c"] Oct 30 17:01:52.467846 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.467830 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2"] Oct 30 17:01:52.468576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.467866 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.555802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.555765 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.642945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.642781 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2"] Oct 30 17:01:52.661562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661064 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.661562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661105 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.661562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661202 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc85k\" (UniqueName: \"kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-kube-api-access-mc85k\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.661562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661255 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.661562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661303 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.661562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661431 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.661562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661475 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.662040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661572 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.662040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661606 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.662040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661644 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.662040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661690 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.662040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661722 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.662040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661774 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.662040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661812 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.662040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661839 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.662040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661949 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.662040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661972 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.662040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.661990 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npphz\" (UniqueName: \"kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-kube-api-access-npphz\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.667460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.667439 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk"] Oct 30 17:01:52.751311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.751226 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.753597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.753492 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk"] Oct 30 17:01:52.763506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.763478 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.763634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.763598 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.763699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.763653 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.763753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.763711 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.763753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.763744 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.763864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.763793 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.763864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.763825 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.764004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.763882 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.764004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.763970 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.764121 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.764059 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.764175 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.764118 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.764231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.764196 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.764231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.764202 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.764336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.764262 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.764336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.764294 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.764436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.764341 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npphz\" (UniqueName: \"kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-kube-api-access-npphz\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.764436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.764373 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvzfp\" (UniqueName: \"kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-kube-api-access-qvzfp\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.765579 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.764507 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.765685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.764739 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-credential-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.765749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.765697 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.765749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.765725 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.765880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.765819 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc85k\" (UniqueName: \"kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-kube-api-access-mc85k\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.765880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.765849 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.765880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.765876 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3df5215-60e4-4c09-904e-b891da8ee19f-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.766054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.765944 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.766054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.765978 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.766054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.766005 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.766217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.766092 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.766217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.766096 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istiod-ca-cert\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.766217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.766131 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.766483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.766460 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.766576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.766529 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-socket\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.766950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.766914 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.767225 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.767191 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.767459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.767435 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-certs\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.786924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.768965 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.786924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.772466 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.826148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.826110 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-data\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.843577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.843466 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.847612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.846415 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-podinfo\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.852941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.849476 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-envoy\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.852941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.852577 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.853678 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.853651 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npphz\" (UniqueName: \"kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-kube-api-access-npphz\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.854181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.854158 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-token\") pod \"e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.855601 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.855530 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc85k\" (UniqueName: \"kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-kube-api-access-mc85k\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.930545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.869665 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvzfp\" (UniqueName: \"kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-kube-api-access-qvzfp\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.930545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.869757 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.930545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.869836 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.930545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.869863 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3df5215-60e4-4c09-904e-b891da8ee19f-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.930545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.869911 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.930545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.870002 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.930545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.870045 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.930545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.870095 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.930545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.870162 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.930545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.870562 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.930545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.886332 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.930545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.886594 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.930545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.928347 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.942507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.929037 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3df5215-60e4-4c09-904e-b891da8ee19f-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.947184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.946711 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.950929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.950768 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.951110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.951071 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:52.954473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.954010 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:52.958611 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.958589 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:52.963105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:52.963083 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvzfp\" (UniqueName: \"kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-kube-api-access-qvzfp\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:53.161925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:53.159548 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:53.181089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:53.174380 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:01:53.181089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:53.174445 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:01:53.246778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:53.246227 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:01:53.246778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:53.246286 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:01:53.756393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:53.755953 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2"] Oct 30 17:01:53.757303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:53.757206 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c"] Oct 30 17:01:53.761260 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:53.760416 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:53.761260 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:53.760501 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:53.761260 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:53.760547 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:53.768111 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:53.768067 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:53.768219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:53.768167 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:53.768219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:53.768212 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:54.028722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.028679 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk"] Oct 30 17:01:54.052449 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:01:54.052399 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3df5215_60e4_4c09_904e_b891da8ee19f.slice/crio-8749e85069b7e909827a1e7f671f8e452b3fa43ea98849ef8783a75743dc9678 WatchSource:0}: Error finding container 8749e85069b7e909827a1e7f671f8e452b3fa43ea98849ef8783a75743dc9678: Status 404 returned error can't find the container with id 8749e85069b7e909827a1e7f671f8e452b3fa43ea98849ef8783a75743dc9678 Oct 30 17:01:54.055657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.055452 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:54.055657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.055535 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:54.055657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.055580 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:01:54.169075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.164285 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:01:54.169075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.164344 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:01:54.243169 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.243044 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:01:54.243169 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.243099 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:01:54.391884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.391805 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" event={"ID":"f3df5215-60e4-4c09-904e-b891da8ee19f","Type":"ContainerStarted","Data":"8749e85069b7e909827a1e7f671f8e452b3fa43ea98849ef8783a75743dc9678"} Oct 30 17:01:54.426883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.426749 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" event={"ID":"f5d62d2b-8034-4de7-a7f4-a8e8f6974679","Type":"ContainerStarted","Data":"91836df686bcd8c82b6e413ecd67633604b87ad46e44334a8730d8c23078d275"} Oct 30 17:01:54.426883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.426817 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" event={"ID":"f5d62d2b-8034-4de7-a7f4-a8e8f6974679","Type":"ContainerStarted","Data":"0715873e285e4e7ab919fdc5dfe5d235dd4555009e5000a813ac87c2a8e78ffc"} Oct 30 17:01:54.475974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.475278 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:54.477250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.477217 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" event={"ID":"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542","Type":"ContainerStarted","Data":"f6e343f433043cdd5c45db77d5a482f88b2c6335b55db60a8d8e30f309248897"} Oct 30 17:01:54.510283 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.510250 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" event={"ID":"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542","Type":"ContainerStarted","Data":"8d12196d6e16f0da6bbb7d4a52d4cb66414ca7a3cc82c3804c771760ef9459ac"} Oct 30 17:01:54.511424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.510029 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podStartSLOduration=2.5100045399999997 podStartE2EDuration="2.51000454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:54.480269894 +0000 UTC m=+892.125780074" watchObservedRunningTime="2026-10-30 17:01:54.51000454 +0000 UTC m=+892.155514720" Oct 30 17:01:54.607106 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.600654 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podStartSLOduration=3.6006333379999997 podStartE2EDuration="3.600633338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:54.540991549 +0000 UTC m=+892.186501732" watchObservedRunningTime="2026-10-30 17:01:54.600633338 +0000 UTC m=+892.246143518" Oct 30 17:01:54.874322 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.834885 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:54.963922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.957241 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:01:54.986253 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:54.986217 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:01:55.163974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:55.163512 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:01:55.163974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:55.163599 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:01:55.245415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:55.245375 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:01:55.245703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:55.245673 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:01:55.543926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:55.543157 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" event={"ID":"f3df5215-60e4-4c09-904e-b891da8ee19f","Type":"ContainerStarted","Data":"3b8dc5a708fbd6075978e0d343b6ef59271d95877bd04a983496b18a0cacfab8"} Oct 30 17:01:55.634926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:55.627178 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podStartSLOduration=3.627158919 podStartE2EDuration="3.627158919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:55.622226871 +0000 UTC m=+893.267737052" watchObservedRunningTime="2026-10-30 17:01:55.627158919 +0000 UTC m=+893.272669099" Oct 30 17:01:55.990985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:55.976029 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:01:55.990985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:55.976099 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 17:01:55.990985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:55.976449 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:01:55.990985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:55.976505 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:01:56.171926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:56.171823 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:01:56.172106 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:56.171921 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:01:56.175922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:56.172339 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:01:56.175922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:56.172470 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:01:56.175922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:56.172505 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 17:01:56.240924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:56.240767 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:01:56.240924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:56.240845 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:01:57.057925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.052381 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:01:57.057925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.052471 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:01:57.090129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.090096 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:01:57.090428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.090394 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 17:01:57.229924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.186121 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:01:57.229924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.186181 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:01:57.345493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.256048 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:01:57.345493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.256118 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:01:57.367473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.367438 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:01:57.367728 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.367700 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 17:01:57.980978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.956354 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:01:57.981379 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.981221 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:01:57.981582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.976354 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:01:57.981726 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.981706 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 17:01:57.981850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:57.975995 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:01:58.055189 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:58.044205 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:58.167543 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:58.167505 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:01:58.181432 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:58.181394 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 17:01:58.213046 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:58.181036 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:01:58.213357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:58.213323 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:01:58.306926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:58.293933 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:01:58.306926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:58.294176 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:01:58.959527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:58.959485 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:01:58.959947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:58.959891 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:01:58.960497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:58.959833 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:01:58.965927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:58.960618 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 17:01:59.068629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:59.068593 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:01:59.205930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:59.205330 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:01:59.205930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:59.205394 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:01:59.236141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:59.209061 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:01:59.236141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:59.209125 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 17:01:59.256921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:59.249025 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:01:59.256921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:59.249110 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:01:59.957376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:59.957328 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:01:59.999939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:01:59.957651 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:02:00.030506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:00.030466 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:02:00.030790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:00.030761 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 17:02:00.167472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:00.167438 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:02:00.167782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:00.167740 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 17:02:00.168481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:00.168448 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:02:00.168632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:00.168611 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:02:00.260146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:00.260112 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:02:00.261738 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:00.261700 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:02:00.964615 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:00.964567 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:02:00.964894 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:00.964852 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:02:00.965731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:00.965695 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:02:00.965891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:00.965871 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 17:02:01.024943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:01.023740 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:01.164351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:01.164311 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:02:01.164614 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:01.164585 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 17:02:01.165117 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:01.164320 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:02:01.165301 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:01.165281 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:02:01.247599 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:01.247482 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:02:01.248030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:01.247984 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:02:01.979841 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:01.955280 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:02:01.979841 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:01.955345 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:02:01.980931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:01.980718 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:02:01.980931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:01.980774 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 17:02:02.171988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:02.161294 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:02:02.199925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:02.175979 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 17:02:02.200822 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:02.171817 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:02:02.222123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:02.222062 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:02:02.242120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:02.240054 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:02:02.242120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:02.240120 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:02:03.016938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.014349 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:02:03.060168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.021041 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 17:02:03.060168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.017967 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:02:03.060168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.021678 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:02:03.258929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.255672 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:02:03.275878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.275785 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 17:02:03.285925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.268026 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 17:02:03.313924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.313031 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 17:02:03.313924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.274547 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 17:02:03.313924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.313132 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 17:02:03.645997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.623007 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:03.960982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.960581 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:02:03.960982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.960680 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:02:03.960982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.960793 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:02:03.960982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:03.960818 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 17:02:04.191927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:04.178168 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:02:04.191927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:04.179495 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:02:04.191927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:04.179551 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 17:02:04.259985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:04.256124 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:02:04.852925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:04.849443 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:02:04.852925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:04.849662 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:02:04.957924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:04.957350 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:02:04.957924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:04.957415 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:02:04.964925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:04.961266 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:02:04.964925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:04.961322 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 17:02:05.163976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:05.163447 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:02:05.163976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:05.163512 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 17:02:05.215924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:05.215285 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:02:05.215924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:05.215367 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:05.303927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:05.299087 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:02:05.303927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:05.299170 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:05.816791 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:05.816754 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:05.959535 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:05.959496 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:02:05.959822 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:05.959791 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:02:05.960387 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:05.960361 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:02:05.960541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:05.960519 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 17:02:06.190097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:06.190002 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:02:06.190336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:06.190307 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 17:02:06.260012 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:06.259975 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:02:06.260609 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:06.260579 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:06.305846 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:06.304227 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:02:06.310172 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:06.310140 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:02:06.437985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:06.437936 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:06.779933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:06.776515 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:02:06.815306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:06.815276 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:02:06.965939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:06.964156 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:02:06.965939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:06.964242 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 17:02:06.965939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:06.965167 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:02:06.965939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:06.965219 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:02:07.201971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.198509 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 17:02:07.201971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.198567 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 17:02:07.253173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.253135 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5"] Oct 30 17:02:07.292059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.292027 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.351063 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.351034 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5"] Oct 30 17:02:07.437373 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.414705 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.438864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.438831 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.439177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.439154 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.439316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.439299 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.439453 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.439437 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce315878-d81e-42b8-89d5-26086c747168-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.439595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.439580 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce315878-d81e-42b8-89d5-26086c747168-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.439736 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.439721 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.439998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.439978 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfdkq\" (UniqueName: \"kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-kube-api-access-dfdkq\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.440226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.440206 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.542562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.542022 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.543011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.542988 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.543623 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.543605 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.544107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.544089 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.544939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.544920 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce315878-d81e-42b8-89d5-26086c747168-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.564923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.545608 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce315878-d81e-42b8-89d5-26086c747168-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.566296 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.566266 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.566600 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.566579 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfdkq\" (UniqueName: \"kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-kube-api-access-dfdkq\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.566821 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.566805 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.567059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.543513 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.567196 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.544840 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.567612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.567590 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.567734 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.544017 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.568137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.550574 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce315878-d81e-42b8-89d5-26086c747168-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.598001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.589543 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce315878-d81e-42b8-89d5-26086c747168-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.642857 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.615961 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.642857 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.637098 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.663920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.660541 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfdkq\" (UniqueName: \"kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-kube-api-access-dfdkq\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.770792 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.770756 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:07.985935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.958306 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 17:02:07.985935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.958365 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 17:02:07.985935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.958771 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:02:07.985935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.959151 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:02:07.985935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.972017 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 17:02:07.985935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:07.972078 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 17:02:08.164101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:08.163329 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:02:08.346613 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:02:08.346544 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:02:08.469727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:08.469626 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:08.570994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:08.563871 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5"] Oct 30 17:02:08.739961 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:02:08.737693 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce315878_d81e_42b8_89d5_26086c747168.slice/crio-07075faa715ad2d960d9658385b66b5db895f0e3af3c20e68890cc58521c88d5 WatchSource:0}: Error finding container 07075faa715ad2d960d9658385b66b5db895f0e3af3c20e68890cc58521c88d5: Status 404 returned error can't find the container with id 07075faa715ad2d960d9658385b66b5db895f0e3af3c20e68890cc58521c88d5 Oct 30 17:02:08.832276 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:08.832237 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:08.832527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:08.832505 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:08.832672 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:08.832652 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:09.048980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:09.039744 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" event={"ID":"ce315878-d81e-42b8-89d5-26086c747168","Type":"ContainerStarted","Data":"07075faa715ad2d960d9658385b66b5db895f0e3af3c20e68890cc58521c88d5"} Oct 30 17:02:09.185851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:09.185175 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:02:09.185851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:09.185259 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:09.974059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:09.974017 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:02:09.974694 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:09.974659 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:10.027080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.027036 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:02:10.027366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.027331 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:10.159917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.159860 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" event={"ID":"ce315878-d81e-42b8-89d5-26086c747168","Type":"ContainerStarted","Data":"2feac357f730725672e7c5f6aa443ae8b6836c350c7031592120df5b6014af02"} Oct 30 17:02:10.225368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.225254 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" podStartSLOduration=3.225234295 podStartE2EDuration="3.225234295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:10.222939395 +0000 UTC m=+907.868449573" watchObservedRunningTime="2026-10-30 17:02:10.225234295 +0000 UTC m=+907.870744478" Oct 30 17:02:10.241962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.227978 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:02:10.241962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.228050 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:10.241962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.226973 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:02:10.247491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.247462 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:02:10.795115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.794635 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:10.795115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.794752 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:02:10.795115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.794796 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" podUID="ce315878-d81e-42b8-89d5-26086c747168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 17:02:10.852003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.851874 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:10.895936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.893268 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:02:10.901936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:10.901214 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:02:11.032924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:11.028031 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:02:11.032924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:11.028104 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:11.731634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:11.731580 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:02:11.733679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:11.733622 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:02:11.791440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:11.791403 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:02:11.791732 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:11.791704 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" podUID="ce315878-d81e-42b8-89d5-26086c747168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 17:02:12.791924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:12.791296 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:02:12.791924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:12.791360 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" podUID="ce315878-d81e-42b8-89d5-26086c747168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 17:02:12.912926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:12.909773 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:13.800047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:13.792975 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:02:13.800047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:13.793056 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" podUID="ce315878-d81e-42b8-89d5-26086c747168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 17:02:13.802100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:13.802065 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:14.727185 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:14.727063 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:14.802926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:14.794990 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:02:14.802926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:14.795092 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" podUID="ce315878-d81e-42b8-89d5-26086c747168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 17:02:15.794924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:15.794476 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:02:15.794924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:15.794537 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" podUID="ce315878-d81e-42b8-89d5-26086c747168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 17:02:16.668128 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:16.668088 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:16.790922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:16.790208 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 17:02:16.790922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:16.790258 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" podUID="ce315878-d81e-42b8-89d5-26086c747168" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 17:02:17.794500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:17.794464 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:17.795076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:17.795003 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:17.795821 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:17.795794 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:02:18.395925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:18.388582 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:20.095924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:20.095520 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:21.706926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:21.695807 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:23.158598 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.158551 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh"] Oct 30 17:02:23.305866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.305731 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh"] Oct 30 17:02:23.306679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.306656 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.318662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.318607 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.318953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.318932 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.319102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.319087 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69zc5\" (UniqueName: \"kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-kube-api-access-69zc5\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.319228 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.319215 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.319340 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.319328 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.319500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.319484 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.319645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.319631 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.319809 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.319795 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.319923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.319892 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.422503 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.422401 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69zc5\" (UniqueName: \"kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-kube-api-access-69zc5\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.422783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.422754 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.422972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.422954 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.423193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.423173 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.423370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.423353 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.423563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.423544 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.423688 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.423670 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.423978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.423953 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.432230 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.432189 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.436811 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.428167 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.437482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.425083 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.437679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.430226 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.437809 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.431653 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.470697 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.429971 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.490846 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.490813 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.492980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.492925 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.534029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.533996 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69zc5\" (UniqueName: \"kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-kube-api-access-69zc5\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.551922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.546338 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:23.649640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:23.649591 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:24.217147 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:24.216935 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:24.660448 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:02:24.660291 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4546e53a_e9f2_4e09_ad63_c9df1bb57570.slice/crio-cc0e7b3bc86a413a9c4e39bd3905499d10dddb23f53fafb1b1139096108a69b0 WatchSource:0}: Error finding container cc0e7b3bc86a413a9c4e39bd3905499d10dddb23f53fafb1b1139096108a69b0: Status 404 returned error can't find the container with id cc0e7b3bc86a413a9c4e39bd3905499d10dddb23f53fafb1b1139096108a69b0 Oct 30 17:02:24.660448 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:24.660361 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh"] Oct 30 17:02:24.663512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:24.663479 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:24.663605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:24.663546 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:24.663605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:24.663577 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:25.284544 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:25.284507 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:25.578985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:25.578646 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" event={"ID":"4546e53a-e9f2-4e09-ad63-c9df1bb57570","Type":"ContainerStarted","Data":"a47fbe97f36fd01e63534a593c45ee452441fd2dec4051b9ccad7bbc01dbc848"} Oct 30 17:02:25.578985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:25.578719 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" event={"ID":"4546e53a-e9f2-4e09-ad63-c9df1bb57570","Type":"ContainerStarted","Data":"cc0e7b3bc86a413a9c4e39bd3905499d10dddb23f53fafb1b1139096108a69b0"} Oct 30 17:02:25.650457 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:25.650426 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:25.652003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:25.651977 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 17:02:25.652138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:25.652031 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" podUID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 17:02:25.656035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:25.655988 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" podStartSLOduration=3.655976055 podStartE2EDuration="3.655976055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:25.640378805 +0000 UTC m=+923.285888984" watchObservedRunningTime="2026-10-30 17:02:25.655976055 +0000 UTC m=+923.301486235" Oct 30 17:02:25.924619 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:25.924529 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:26.650831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:26.650790 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 17:02:26.651211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:26.650847 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" podUID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 17:02:27.651357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:27.651189 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 17:02:27.651357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:27.651250 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" podUID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 17:02:27.704744 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:27.704705 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:28.650747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:28.650711 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 17:02:28.650952 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:28.650770 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" podUID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 17:02:29.608496 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.608459 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw"] Oct 30 17:02:29.628266 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.628237 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw"] Oct 30 17:02:29.628590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.628347 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.652724 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.652691 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 17:02:29.652886 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.652753 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" podUID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 17:02:29.659132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.659098 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-096d75\"/\"distinct-2-w0-096d75-kserve-self-signed-certs\"" Oct 30 17:02:29.659384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.659157 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-096d75\"/\"openshift-service-ca.crt\"" Oct 30 17:02:29.659384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.659244 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-096d75\"/\"default-dockercfg-m777j\"" Oct 30 17:02:29.659616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.659599 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-096d75\"/\"kube-root-ca.crt\"" Oct 30 17:02:29.714183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.714149 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:29.758789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.758749 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-home\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.758988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.758932 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bpv4\" (UniqueName: \"kubernetes.io/projected/31569569-b53b-43c8-a6ca-2970efc5e331-kube-api-access-6bpv4\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.758988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.758965 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31569569-b53b-43c8-a6ca-2970efc5e331-tls-certs\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.759118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.759093 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-dshm\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.759157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.759132 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-tmp-dir\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.759251 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.759235 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-model-cache\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.867975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.865045 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bpv4\" (UniqueName: \"kubernetes.io/projected/31569569-b53b-43c8-a6ca-2970efc5e331-kube-api-access-6bpv4\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.867975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.865109 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31569569-b53b-43c8-a6ca-2970efc5e331-tls-certs\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.867975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.865206 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-dshm\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.867975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.865238 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-tmp-dir\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.867975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.865317 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-model-cache\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.867975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.865409 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-home\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.867975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.866077 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-home\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.889364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.889327 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-tmp-dir\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.889865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.889841 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-model-cache\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.894109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.894077 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-dshm\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:29.923812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:29.923777 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:30.111912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:30.111832 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31569569-b53b-43c8-a6ca-2970efc5e331-tls-certs\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:30.113970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:30.113941 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bpv4\" (UniqueName: \"kubernetes.io/projected/31569569-b53b-43c8-a6ca-2970efc5e331-kube-api-access-6bpv4\") pod \"distinct-2-w0-096d75-kserve-6c7d886775-5m9cw\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:30.245382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:30.245277 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:30.605274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:30.605242 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw"] Oct 30 17:02:30.606407 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:02:30.606378 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31569569_b53b_43c8_a6ca_2970efc5e331.slice/crio-3c372a8af3e5e59fdb452cca7bf64594087c7d0e2819adc7d762c701ef599c93 WatchSource:0}: Error finding container 3c372a8af3e5e59fdb452cca7bf64594087c7d0e2819adc7d762c701ef599c93: Status 404 returned error can't find the container with id 3c372a8af3e5e59fdb452cca7bf64594087c7d0e2819adc7d762c701ef599c93 Oct 30 17:02:30.650256 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:30.650209 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 17:02:30.650653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:30.650280 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" podUID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 17:02:30.650817 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:30.650793 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" event={"ID":"31569569-b53b-43c8-a6ca-2970efc5e331","Type":"ContainerStarted","Data":"3c372a8af3e5e59fdb452cca7bf64594087c7d0e2819adc7d762c701ef599c93"} Oct 30 17:02:31.650611 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:31.650570 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 17:02:31.651120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:31.650657 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" podUID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 17:02:31.668431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:31.668394 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:31.710223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:31.704439 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" event={"ID":"31569569-b53b-43c8-a6ca-2970efc5e331","Type":"ContainerStarted","Data":"ab32c091c022325c2c6d3f1b974722dcaafb9e6f4deff6eba89e0b4bbe2d52d1"} Oct 30 17:02:31.777056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:31.776942 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podStartSLOduration=2.776917295 podStartE2EDuration="2.776917295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:31.772076593 +0000 UTC m=+929.417586786" watchObservedRunningTime="2026-10-30 17:02:31.776917295 +0000 UTC m=+929.422427473" Oct 30 17:02:32.245611 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:32.245573 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:32.656709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:32.656677 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:32.657226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:32.657215 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:32.658239 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:32.658217 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:02:33.520933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:33.520857 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:35.252369 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:35.252320 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:36.884197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:36.884159 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:37.913304 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:02:37.913263 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:02:38.065138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.065108 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw"] Oct 30 17:02:38.069618 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.069594 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.135714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.133998 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw"] Oct 30 17:02:38.249149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.249056 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.249149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.249120 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.249376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.249173 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.249376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.249204 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.249490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.249388 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.249546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.249489 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.249546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.249511 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.249546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.249526 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.249706 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.249559 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kjlp\" (UniqueName: \"kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-kube-api-access-9kjlp\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.350497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.350463 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.350671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.350539 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.350671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.350567 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.350671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.350588 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.350671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.350638 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kjlp\" (UniqueName: \"kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-kube-api-access-9kjlp\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.350884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.350735 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.350884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.350767 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.350884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.350814 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.350884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.350861 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.351123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.350976 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-data\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.351123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.350997 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-certs\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.351235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.351141 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-credential-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.351426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.351404 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-socket\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.351500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.351428 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istiod-ca-cert\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.353153 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.353127 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-podinfo\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.353248 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.353132 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-envoy\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.377523 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.377495 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-token\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.377827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.377801 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kjlp\" (UniqueName: \"kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-kube-api-access-9kjlp\") pod \"e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.418108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.418076 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:38.696944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.692883 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:38.778326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.769861 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw"] Oct 30 17:02:38.787575 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.787533 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:38.787702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.787626 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:38.787702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:38.787669 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:39.806491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:39.806446 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" event={"ID":"1fa37ccf-2a69-4083-a6db-b32cd13132a4","Type":"ContainerStarted","Data":"1c3f9ef40f2bf401e648b20c0b5b472c1735b410e843f94ad0b1104f3d394129"} Oct 30 17:02:39.806997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:39.806521 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" event={"ID":"1fa37ccf-2a69-4083-a6db-b32cd13132a4","Type":"ContainerStarted","Data":"065aa482507d73100954a773615bf59beb6a467a382871d738f6af2f35941ca4"} Oct 30 17:02:39.908726 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:39.908657 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" podStartSLOduration=1.9086383 podStartE2EDuration="1.9086383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:39.877389224 +0000 UTC m=+937.522899404" watchObservedRunningTime="2026-10-30 17:02:39.9086383 +0000 UTC m=+937.554148484" Oct 30 17:02:40.253979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:40.253411 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:40.425921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:40.423769 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 17:02:40.425921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:40.423828 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" podUID="1fa37ccf-2a69-4083-a6db-b32cd13132a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 17:02:40.425921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:40.424604 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:40.433925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:40.427787 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:40.634926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:40.629496 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:41.368683 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:41.368385 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:02:41.425619 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:41.425585 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 17:02:41.425916 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:41.425871 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" podUID="1fa37ccf-2a69-4083-a6db-b32cd13132a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 17:02:42.448930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:42.442722 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 17:02:42.448930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:42.442795 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" podUID="1fa37ccf-2a69-4083-a6db-b32cd13132a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 17:02:42.923923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:42.914612 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:43.419275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:43.419241 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 17:02:43.419503 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:43.419483 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" podUID="1fa37ccf-2a69-4083-a6db-b32cd13132a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 17:02:44.418939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.418865 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 17:02:44.419370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.418969 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" podUID="1fa37ccf-2a69-4083-a6db-b32cd13132a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 17:02:44.601937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.601887 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p"] Oct 30 17:02:44.607042 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.607012 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.624952 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.623002 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b80d83\"/\"kube-root-ca.crt\"" Oct 30 17:02:44.669994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.669317 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b80d83\"/\"distinct-2-w3-b80d83-kserve-self-signed-certs\"" Oct 30 17:02:44.669994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.669398 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b80d83\"/\"openshift-service-ca.crt\"" Oct 30 17:02:44.669994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.669518 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b80d83\"/\"default-dockercfg-pnr6k\"" Oct 30 17:02:44.728112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.712283 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-dshm\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.728112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.712348 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-model-cache\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.728112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.712478 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bddf0988-e42b-43c1-80a8-e22d04bdb133-tls-certs\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.728112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.712548 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-home\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.728112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.712572 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8sr4\" (UniqueName: \"kubernetes.io/projected/bddf0988-e42b-43c1-80a8-e22d04bdb133-kube-api-access-j8sr4\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.728112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.712640 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-tmp-dir\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.730250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.729178 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p"] Oct 30 17:02:44.730250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.730149 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:44.837924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.837784 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bddf0988-e42b-43c1-80a8-e22d04bdb133-tls-certs\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.837924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.837881 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-home\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.838174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.837955 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8sr4\" (UniqueName: \"kubernetes.io/projected/bddf0988-e42b-43c1-80a8-e22d04bdb133-kube-api-access-j8sr4\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.838174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.838026 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-tmp-dir\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.838174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.838120 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-dshm\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.838174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.838165 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-model-cache\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.843919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.838715 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-model-cache\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.857041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.857007 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bddf0988-e42b-43c1-80a8-e22d04bdb133-tls-certs\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.863997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.857522 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-home\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.864245 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.859595 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-tmp-dir\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.886762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.886724 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:44.914925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.914661 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8sr4\" (UniqueName: \"kubernetes.io/projected/bddf0988-e42b-43c1-80a8-e22d04bdb133-kube-api-access-j8sr4\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:44.950372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:44.950247 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-dshm\") pod \"distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:45.224227 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:45.224147 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:45.419051 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:45.419013 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 17:02:45.419456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:45.419080 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" podUID="1fa37ccf-2a69-4083-a6db-b32cd13132a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 17:02:45.557122 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:02:45.557093 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbddf0988_e42b_43c1_80a8_e22d04bdb133.slice/crio-2abeb0faddcb3fe1a60c37d7ce5e9c5d8d430e15958aba028b1b0893b3564646 WatchSource:0}: Error finding container 2abeb0faddcb3fe1a60c37d7ce5e9c5d8d430e15958aba028b1b0893b3564646: Status 404 returned error can't find the container with id 2abeb0faddcb3fe1a60c37d7ce5e9c5d8d430e15958aba028b1b0893b3564646 Oct 30 17:02:45.557601 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:45.557571 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p"] Oct 30 17:02:46.229716 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:46.229668 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" event={"ID":"bddf0988-e42b-43c1-80a8-e22d04bdb133","Type":"ContainerStarted","Data":"a0eaa42278795a387e4c1a26b484906aa6b0c214823d248c8a2191d1668b4fb8"} Oct 30 17:02:46.229935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:46.229735 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" event={"ID":"bddf0988-e42b-43c1-80a8-e22d04bdb133","Type":"ContainerStarted","Data":"2abeb0faddcb3fe1a60c37d7ce5e9c5d8d430e15958aba028b1b0893b3564646"} Oct 30 17:02:46.275909 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:46.275836 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podStartSLOduration=2.27582017 podStartE2EDuration="2.27582017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:46.273434812 +0000 UTC m=+943.918944995" watchObservedRunningTime="2026-10-30 17:02:46.27582017 +0000 UTC m=+943.921330357" Oct 30 17:02:46.808047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:46.808008 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:47.225092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:47.225008 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:47.427420 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:47.427286 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:02:47.427420 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:47.427360 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" podUID="1fa37ccf-2a69-4083-a6db-b32cd13132a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:48.231933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:48.231868 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:48.232792 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:48.232764 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:48.234633 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:48.234609 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:02:48.444071 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:48.444014 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:50.039686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:50.039645 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:51.768938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:51.768881 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:52.833840 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:52.833805 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl"] Oct 30 17:02:52.838418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:52.838394 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:52.878312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:52.878279 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl"] Oct 30 17:02:52.910615 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:52.910570 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88601bd-82a5-47c7-be65-67915627db91-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:52.910810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:52.910717 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88601bd-82a5-47c7-be65-67915627db91-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:52.910810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:52.910764 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:52.910956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:52.910886 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:52.911015 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:52.910966 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:52.911067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:52.911052 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:52.911121 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:52.911103 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh92c\" (UniqueName: \"kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-kube-api-access-fh92c\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:52.911230 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:52.911214 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:52.911287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:52.911250 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.012148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.012108 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88601bd-82a5-47c7-be65-67915627db91-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.012359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.012187 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88601bd-82a5-47c7-be65-67915627db91-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.012359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.012214 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.012359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.012260 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.012359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.012290 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.012577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.012398 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.012577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.012429 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh92c\" (UniqueName: \"kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-kube-api-access-fh92c\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.012577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.012454 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.012577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.012482 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.012776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.012644 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.012957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.012931 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.013091 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.012975 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88601bd-82a5-47c7-be65-67915627db91-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.013159 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.013091 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.013262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.013233 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.014942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.014878 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.015097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.015079 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88601bd-82a5-47c7-be65-67915627db91-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.037332 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.037298 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.037552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.037534 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh92c\" (UniqueName: \"kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-kube-api-access-fh92c\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.153390 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.153298 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:53.424611 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.424520 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:53.541467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.541435 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl"] Oct 30 17:02:53.542244 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.542205 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:53.542349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.542270 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:53.542349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:53.542301 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:02:54.313077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:54.313033 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" event={"ID":"e88601bd-82a5-47c7-be65-67915627db91","Type":"ContainerStarted","Data":"4c747700874f5ff6f4905b4afca2cfd882305632ec2cd42befdb1b3dc7da4f07"} Oct 30 17:02:54.313559 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:54.313105 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" event={"ID":"e88601bd-82a5-47c7-be65-67915627db91","Type":"ContainerStarted","Data":"3197dbd5fcd51a72ddd2e1c8ab08b5771ddd73d8b9d24df042e035382792ef3a"} Oct 30 17:02:54.399934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:54.398959 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" podStartSLOduration=2.398937996 podStartE2EDuration="2.398937996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:54.374420109 +0000 UTC m=+952.019930291" watchObservedRunningTime="2026-10-30 17:02:54.398937996 +0000 UTC m=+952.044448180" Oct 30 17:02:54.427884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:54.427847 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:55.167924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:55.158756 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:02:55.167924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:55.158891 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 17:02:55.167924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:55.158953 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" podUID="e88601bd-82a5-47c7-be65-67915627db91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 17:02:55.232922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:55.231922 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:55.427003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:55.423597 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:55.427003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:55.424116 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:55.450557 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:55.450527 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:02:55.597651 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:55.597446 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:56.154419 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:56.154381 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 17:02:56.154690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:56.154659 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" podUID="e88601bd-82a5-47c7-be65-67915627db91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 17:02:57.244726 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:57.244682 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 17:02:57.245261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:57.244755 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" podUID="e88601bd-82a5-47c7-be65-67915627db91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 17:02:57.809378 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:57.809322 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:02:58.153891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:58.153810 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 17:02:58.154065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:58.153883 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" podUID="e88601bd-82a5-47c7-be65-67915627db91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 17:02:59.154341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:59.154309 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 17:02:59.154830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:59.154366 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" podUID="e88601bd-82a5-47c7-be65-67915627db91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 17:02:59.764921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:02:59.757595 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:00.153807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:00.153727 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 17:03:00.153807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:00.153781 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" podUID="e88601bd-82a5-47c7-be65-67915627db91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 17:03:01.158840 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:01.158808 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:03:01.159292 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:01.159242 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:03:01.160061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:01.160040 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:03:01.367433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:01.367398 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:02.995062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:02.995015 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:04.589660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:04.589603 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:06.184119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:06.184084 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:07.236942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.236871 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh"] Oct 30 17:03:07.243189 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.243166 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.281198 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.281165 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh"] Oct 30 17:03:07.281381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.281347 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.281460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.281403 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.281460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.281447 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.281537 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.281477 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.281570 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.281548 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.281620 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.281589 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.281683 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.281658 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.281739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.281696 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scsm4\" (UniqueName: \"kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-kube-api-access-scsm4\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.281817 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.281799 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.383459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.383421 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.383661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.383497 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.383661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.383524 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scsm4\" (UniqueName: \"kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-kube-api-access-scsm4\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.383661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.383603 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.383827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.383677 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.383827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.383701 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.383827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.383729 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.383827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.383748 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.383827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.383803 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.384200 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.384178 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.384434 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.384411 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.385070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.385048 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.386038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.385383 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.386038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.385818 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.387824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.387798 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.388153 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.388110 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.411143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.411104 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.411280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.411205 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scsm4\" (UniqueName: \"kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-kube-api-access-scsm4\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.558374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.558334 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:07.946166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.946023 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:07.969523 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.969477 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh"] Oct 30 17:03:07.973894 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:03:07.973804 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3108a244_2537_47cb_a2dd_01b73c5cd3e9.slice/crio-b26e083c56d8a013bf71b15ab173ba936dd810587b72f89bc68ffe881e3d97c2 WatchSource:0}: Error finding container b26e083c56d8a013bf71b15ab173ba936dd810587b72f89bc68ffe881e3d97c2: Status 404 returned error can't find the container with id b26e083c56d8a013bf71b15ab173ba936dd810587b72f89bc68ffe881e3d97c2 Oct 30 17:03:07.976865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.976823 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:07.977003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.976951 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:07.977067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:07.977004 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:07.984182 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:03:07.983599 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:03:08.555533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:08.555489 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" event={"ID":"3108a244-2537-47cb-a2dd-01b73c5cd3e9","Type":"ContainerStarted","Data":"e1b53e79bc8573079b8f86046e1c89115521ea836d97344cce1560e6de85f76c"} Oct 30 17:03:08.556289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:08.556260 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" event={"ID":"3108a244-2537-47cb-a2dd-01b73c5cd3e9","Type":"ContainerStarted","Data":"b26e083c56d8a013bf71b15ab173ba936dd810587b72f89bc68ffe881e3d97c2"} Oct 30 17:03:08.559372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:08.559337 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:08.620379 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:08.620320 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" podStartSLOduration=1.6203037 podStartE2EDuration="1.6203037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:08.616950971 +0000 UTC m=+966.262461145" watchObservedRunningTime="2026-10-30 17:03:08.6203037 +0000 UTC m=+966.265813880" Oct 30 17:03:09.560433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:09.560387 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 17:03:09.560865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:09.560454 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" podUID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 17:03:09.879253 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:09.879163 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:10.559316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:10.559278 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 17:03:10.559512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:10.559347 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" podUID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 17:03:11.507957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:11.507917 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:11.562745 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:11.562713 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 17:03:11.563080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:11.563052 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" podUID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 17:03:12.459209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.459174 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z"] Oct 30 17:03:12.464734 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.464705 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.498820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.497519 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-86f50a\"/\"default-dockercfg-tw9rv\"" Oct 30 17:03:12.498820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.498218 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-86f50a\"/\"openshift-service-ca.crt\"" Oct 30 17:03:12.498820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.498322 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-86f50a\"/\"kube-root-ca.crt\"" Oct 30 17:03:12.498820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.498431 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-86f50a\"/\"distinct-2-w5-86f50a-kserve-self-signed-certs\"" Oct 30 17:03:12.525383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.525350 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-home\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.526083 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.526058 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-dshm\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.526396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.526376 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h64p\" (UniqueName: \"kubernetes.io/projected/2fb327ed-88b8-4f18-81b7-881fc4480618-kube-api-access-4h64p\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.526731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.526704 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb327ed-88b8-4f18-81b7-881fc4480618-tls-certs\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.526977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.526958 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-model-cache\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.527287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.527270 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-tmp-dir\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.538474 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.538299 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z"] Oct 30 17:03:12.559649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.559622 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 17:03:12.559848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.559831 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" podUID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 17:03:12.562594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.562567 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:12.629525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.629484 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h64p\" (UniqueName: \"kubernetes.io/projected/2fb327ed-88b8-4f18-81b7-881fc4480618-kube-api-access-4h64p\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.629735 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.629584 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb327ed-88b8-4f18-81b7-881fc4480618-tls-certs\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.629735 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.629615 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-model-cache\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.629735 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.629673 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-tmp-dir\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.629931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.629794 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-home\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.629931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.629836 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-dshm\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.630136 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.630110 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-tmp-dir\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.630292 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.630153 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-model-cache\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.630365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.630197 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-home\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.632009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.631991 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-dshm\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.632359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.632345 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb327ed-88b8-4f18-81b7-881fc4480618-tls-certs\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.655636 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.655263 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h64p\" (UniqueName: \"kubernetes.io/projected/2fb327ed-88b8-4f18-81b7-881fc4480618-kube-api-access-4h64p\") pod \"distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:12.829214 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:12.829177 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:13.311023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:13.310941 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z"] Oct 30 17:03:13.314176 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:03:13.314148 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fb327ed_88b8_4f18_81b7_881fc4480618.slice/crio-d54089e849ad7b7bc8e37268f22d902881e2dac00abb9ed0981d7d58f2d1e31c WatchSource:0}: Error finding container d54089e849ad7b7bc8e37268f22d902881e2dac00abb9ed0981d7d58f2d1e31c: Status 404 returned error can't find the container with id d54089e849ad7b7bc8e37268f22d902881e2dac00abb9ed0981d7d58f2d1e31c Oct 30 17:03:13.436482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:13.436443 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:13.578983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:13.565180 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 17:03:13.578983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:13.565333 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" podUID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 17:03:13.644226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:13.643662 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" event={"ID":"2fb327ed-88b8-4f18-81b7-881fc4480618","Type":"ContainerStarted","Data":"43568ab0be19ea1ac8aaacc7a72253e196283fc6c64f4248d7a18c7964f7d2e2"} Oct 30 17:03:13.644226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:13.643719 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" event={"ID":"2fb327ed-88b8-4f18-81b7-881fc4480618","Type":"ContainerStarted","Data":"d54089e849ad7b7bc8e37268f22d902881e2dac00abb9ed0981d7d58f2d1e31c"} Oct 30 17:03:13.713605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:13.713544 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podStartSLOduration=1.713524286 podStartE2EDuration="1.713524286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:13.69233087 +0000 UTC m=+971.337841051" watchObservedRunningTime="2026-10-30 17:03:13.713524286 +0000 UTC m=+971.359034466" Oct 30 17:03:13.833025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:13.831117 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:14.559359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:14.559324 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 17:03:14.559542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:14.559382 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" podUID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 17:03:15.285923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:15.279871 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:15.821281 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:15.821250 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:15.823158 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:15.823134 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:15.824453 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:15.824334 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:03:17.864923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:17.863993 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:18.432043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:18.431992 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:03:18.629923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:18.621295 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:18.715103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:18.715012 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:03:18.715384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:18.715351 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" podUID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:20.590858 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:20.590754 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:22.391250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:22.391196 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:22.830350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:22.830318 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:22.843821 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:22.843791 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:22.911429 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:22.911398 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:24.097926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:24.096667 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:25.797925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:25.793966 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:27.684753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:27.684707 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:29.281922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:29.274974 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:30.979526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:30.979486 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:31.368933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:31.367554 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:31.534188 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:31.534110 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:33.830158 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:33.830112 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:33.870688 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:33.870634 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:33.896862 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:33.896821 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:33.932976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:33.932606 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:34.306628 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:34.290380 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:34.450054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:34.449996 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:34.896199 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:34.896153 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:34.915205 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:34.903279 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:03:34.915205 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:34.906330 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:34.915205 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:34.906779 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:03:34.953040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:34.952991 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 17:03:35.067152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:35.067116 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:35.464537 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:35.464482 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:03:35.487935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:35.466430 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:03:35.929593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:35.929535 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:35.957962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:35.952120 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:03:36.088361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:36.088317 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:36.281888 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:36.278668 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:36.282607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:36.282584 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:03:36.317710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:36.317671 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:03:36.330610 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:36.330577 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:03:36.331051 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:36.331032 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:36.669622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:36.669529 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:37.915032 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:03:37.914996 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:03:38.019395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.019360 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t"] Oct 30 17:03:38.025479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.025450 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.055814 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.055776 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t"] Oct 30 17:03:38.114937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.114884 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.115132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.114992 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.115132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.115020 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.115236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.115157 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf16d96-e3f2-4982-92dc-dc1188d30890-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.115283 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.115271 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.115409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.115384 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.115460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.115444 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.115559 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.115545 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.115662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.115647 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zl4v\" (UniqueName: \"kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-kube-api-access-2zl4v\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.217175 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.217076 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zl4v\" (UniqueName: \"kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-kube-api-access-2zl4v\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.217175 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.217138 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.217415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.217185 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.217415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.217214 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.217415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.217254 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf16d96-e3f2-4982-92dc-dc1188d30890-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.217415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.217309 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.217415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.217402 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.217681 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.217433 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.217681 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.217551 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.217964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.217936 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-certs\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.218155 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.218129 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf16d96-e3f2-4982-92dc-dc1188d30890-istiod-ca-cert\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.218238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.218169 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-credential-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.218294 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.218249 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-socket\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.218332 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.218296 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-data\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.220198 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.220170 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-envoy\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.220646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.220622 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-podinfo\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.242529 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.242497 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-token\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.242668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.242578 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zl4v\" (UniqueName: \"kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-kube-api-access-2zl4v\") pod \"e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.374584 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.374546 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:38.380268 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.380231 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:38.799246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.799210 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t"] Oct 30 17:03:38.801917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.801856 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:38.802014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.801993 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:38.802062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:38.802042 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:39.153589 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:39.153513 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" event={"ID":"5cf16d96-e3f2-4982-92dc-dc1188d30890","Type":"ContainerStarted","Data":"42f48b1c105aaa2a258b64c48051c8836d3005379547f24d41ff4033a2e7ee50"} Oct 30 17:03:39.153589 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:39.153560 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" event={"ID":"5cf16d96-e3f2-4982-92dc-dc1188d30890","Type":"ContainerStarted","Data":"b1bba6a327485b1860341ec553f0a3ac04533c881efa1ee59c5d1caa13c3cd1d"} Oct 30 17:03:39.217476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:39.217419 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" podStartSLOduration=2.217404955 podStartE2EDuration="2.217404955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:39.214095243 +0000 UTC m=+996.859605438" watchObservedRunningTime="2026-10-30 17:03:39.217404955 +0000 UTC m=+996.862915134" Oct 30 17:03:39.381602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:39.381565 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:39.383199 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:39.383174 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 17:03:39.383323 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:39.383222 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 17:03:40.085981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:40.085930 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:40.384982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:40.384333 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 17:03:40.384982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:40.384398 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 17:03:41.395927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:41.391368 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 17:03:41.395927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:41.391433 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 17:03:42.135078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:42.133946 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:42.397981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:42.396022 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 17:03:42.397981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:42.396087 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 17:03:43.398939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:43.383662 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 17:03:43.399772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:43.399673 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 17:03:44.104044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:44.092316 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:44.385990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:44.382296 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 17:03:44.385990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:44.382393 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 17:03:45.381682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:45.381634 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 30 17:03:45.382109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:45.381698 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 30 17:03:45.960674 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:45.960633 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:46.521951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:46.521206 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:46.528349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:46.522831 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:46.529501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:46.529471 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:03:47.684543 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:47.684506 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:49.462217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:49.462179 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:51.028981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:51.028949 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:52.399824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.399787 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn"] Oct 30 17:03:52.404427 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.404403 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.444199 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.444160 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn"] Oct 30 17:03:52.589341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.589304 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.589527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.589379 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.589527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.589486 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.589659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.589629 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.589766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.589745 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.589828 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.589802 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.589940 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.589923 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42886d0b-4093-4f40-a8d3-9fc61d25daab-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.589996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.589979 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7cnn\" (UniqueName: \"kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-kube-api-access-c7cnn\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.590053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.590038 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.692100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.692005 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.692244 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.692105 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.692244 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.692143 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.692244 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.692226 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.692386 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.692305 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.692386 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.692345 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.692480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.692411 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42886d0b-4093-4f40-a8d3-9fc61d25daab-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.692480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.692459 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.692583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.692483 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-data\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.692583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.692468 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7cnn\" (UniqueName: \"kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-kube-api-access-c7cnn\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.692583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.692530 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.692739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.692675 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-certs\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.692739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.692708 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-credential-socket\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.693347 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.693317 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42886d0b-4093-4f40-a8d3-9fc61d25daab-istiod-ca-cert\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.694765 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.694744 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-envoy\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.695067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.695045 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-podinfo\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.717194 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.717165 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-token\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.717357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.717244 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7cnn\" (UniqueName: \"kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-kube-api-access-c7cnn\") pod \"e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.735109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.735072 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:52.806923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:52.800781 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:53.163437 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:03:53.163398 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42886d0b_4093_4f40_a8d3_9fc61d25daab.slice/crio-e486477b590cf24cc248f859bd2653a018f373e4f91db4af2d8ff2b9d59f189c WatchSource:0}: Error finding container e486477b590cf24cc248f859bd2653a018f373e4f91db4af2d8ff2b9d59f189c: Status 404 returned error can't find the container with id e486477b590cf24cc248f859bd2653a018f373e4f91db4af2d8ff2b9d59f189c Oct 30 17:03:53.165745 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:53.165678 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn"] Oct 30 17:03:53.166741 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:53.166709 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:53.166861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:53.166785 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:53.166861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:53.166829 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:03:53.449058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:53.440437 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" event={"ID":"42886d0b-4093-4f40-a8d3-9fc61d25daab","Type":"ContainerStarted","Data":"cde8a492a4e059669a42c8d5417cd32e28347e7208d17d1e10892e780b98b797"} Oct 30 17:03:53.449058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:53.440513 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" event={"ID":"42886d0b-4093-4f40-a8d3-9fc61d25daab","Type":"ContainerStarted","Data":"e486477b590cf24cc248f859bd2653a018f373e4f91db4af2d8ff2b9d59f189c"} Oct 30 17:03:53.737995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:53.735559 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:03:54.762927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:54.762194 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:03:54.762927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:54.762259 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 17:03:54.812782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:54.808445 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:54.952141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:54.952029 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:54.956007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:54.955967 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:55.737921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:55.737807 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:03:55.737921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:55.737866 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 17:03:55.982113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:55.982027 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:56.057249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:56.033156 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:03:56.057249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:56.056714 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:03:56.057249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:56.056777 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:03:56.142927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:56.140826 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:56.181924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:56.180092 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" podStartSLOduration=4.180070349 podStartE2EDuration="4.180070349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:53.544857787 +0000 UTC m=+1011.190367970" watchObservedRunningTime="2026-10-30 17:03:56.180070349 +0000 UTC m=+1013.825580531" Oct 30 17:03:56.479985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:56.404096 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:56.532124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:56.450356 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Oct 30 17:03:56.532382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:56.473114 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 30 17:03:56.898923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:56.895095 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:03:56.898923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:56.895177 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 17:03:57.082921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:57.058380 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:57.084071 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:57.072186 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:57.141927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:57.141161 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:57.234484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:57.166678 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:57.510983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:57.465345 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:03:57.644838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:57.598047 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:03:57.682077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:57.682041 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:03:57.892302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:57.876608 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:03:57.916923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:57.892987 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 17:03:57.977925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:57.918881 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:03:58.072093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:57.950318 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:03:58.072093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:58.063991 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:58.129934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:58.118265 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:03:58.157249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:58.157144 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:58.275928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:58.262840 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:03:58.877223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:58.877181 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:03:58.877487 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:58.877455 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 17:03:58.992925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:58.992316 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:03:59.114925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:59.114252 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:03:59.177491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:59.177388 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:03:59.289932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:59.250934 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:03:59.749968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:59.743376 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:03:59.749968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:03:59.743539 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 17:04:00.580501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:00.580449 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:00.757926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:00.757405 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:04:00.757926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:00.757472 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 17:04:01.735819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:01.735783 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:04:01.736261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:01.735839 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 17:04:02.403923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:02.403463 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:02.735832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:02.735755 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:04:02.735832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:02.735804 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 17:04:03.736479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:03.736439 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 30 17:04:03.736888 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:03.736496 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 30 17:04:04.263274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:04.263235 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:04.741294 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:04.741218 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:04:04.741712 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:04.741583 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:04:04.742286 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:04.742267 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:04:05.879736 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:05.878651 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:07.562794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:07.562756 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:07.912068 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:04:07.911974 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:04:09.474694 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:09.474654 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:11.138324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:11.137308 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:12.742252 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:12.742204 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:14.458622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:14.458529 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:16.279144 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:16.277562 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:17.861163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:17.861102 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:20.190925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:20.187558 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:04:21.191186 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:21.191078 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:21.193286 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:21.193207 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:21.255404 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:21.255360 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": context deadline exceeded" Oct 30 17:04:21.312062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:21.312001 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:21.313152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:21.313123 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:04:22.258136 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:22.258069 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:22.272207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:22.272161 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:04:22.315225 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:22.315171 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:22.327097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:22.323067 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:22.327097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:22.324351 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:04:22.327097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:22.325843 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:23.577278 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:23.577224 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:04:23.602603 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:23.602556 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:23.603723 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:23.603689 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:23.838917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:23.654965 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:23.995938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:23.994947 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" podUID="86d11598-b8b5-4653-93ab-438cf26e63eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 17:04:24.583077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:24.583028 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:24.628667 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:24.628592 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:04:24.629202 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:24.629173 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 17:04:24.775922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:24.769095 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:04:24.775922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:24.769183 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:24.909980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:24.804629 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp" podUID="204a77c5-f77b-4842-ae59-cfe3c86cd976" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 17:04:25.009512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:24.903990 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:25.044157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:25.044099 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-78487d6d64-hdjk9" podUID="41a0571b-6e0f-4fa9-bbd3-d0b31614b19b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.51:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:25.057930 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:04:25.050220 4828 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.513s" Oct 30 17:04:25.245989 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:25.245469 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:04:25.290926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:25.221964 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-b8bbs" podUID="177c030c-d783-44ae-92e9-3368ed00ef0d" containerName="proxy" probeResult="failure" output="Get \"https://10.132.0.52:8443/ready\": context deadline exceeded" Oct 30 17:04:25.568311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:25.565887 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:25.664812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:25.664763 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:25.791961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:25.771308 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:04:25.791961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:25.771364 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:04:25.791961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:25.772374 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:04:25.791961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:25.785326 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:04:27.024239 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:27.024041 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:28.794526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:28.794468 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:30.395427 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:30.395386 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:32.173757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:32.173711 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:33.851933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:33.851763 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:35.650048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:35.647241 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:37.422478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:37.422430 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:37.591248 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:37.591197 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:37.748878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:37.748791 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:37.915026 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:04:37.914985 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:04:39.223480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:39.223443 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:41.032786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:41.032585 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:42.581922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:42.578235 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:42.649494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:42.649437 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:42.770089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:42.677526 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:42.886185 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:42.886084 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:04:43.707928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:43.693584 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:43.741977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:43.738462 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:04:43.819631 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:43.800040 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:43.856127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:43.856097 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:04:44.123062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:43.864384 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:44.123813 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:43.901978 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:04:44.126124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:44.126101 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:04:44.575948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:44.568076 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:04:44.732475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:44.719932 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 17:04:44.855991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:44.855860 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:45.176062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:45.175964 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:04:45.582890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:45.577380 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:45.582890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:45.577799 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:04:45.782254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:45.780391 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:45.876990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:45.863466 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:04:46.184922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:46.184776 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:04:46.270923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:46.267397 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:46.422403 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:46.422364 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:04:46.422675 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:46.422645 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" podUID="7beb4126-0039-4932-be23-21d475782c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:46.544094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:46.544049 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 30 17:04:46.623443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:46.589484 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:46.731192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:46.731143 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:46.829030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:46.812803 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 17:04:46.872968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:46.870072 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:04:46.894090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:46.894044 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:04:46.894416 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:46.894349 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" podUID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:47.191950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:47.191834 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:04:47.192219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:47.192178 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:47.192751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:47.192722 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:47.636925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:47.599975 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:47.780929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:47.764163 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:04:47.780929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:47.764229 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:47.781521 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:47.781323 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:04:47.781521 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:47.781429 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" podUID="aab33acc-67cf-4cc4-b180-50cdee10b872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:04:47.836926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:47.831093 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 17:04:47.889981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:47.873879 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:04:48.037249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:48.029405 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:04:48.037249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:48.029670 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:04:48.037249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:48.035328 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:04:48.050330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:48.049500 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:04:48.750867 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:48.750813 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:50.554579 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:50.554531 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:52.152961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:52.152875 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:52.834595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:52.834532 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:54.061538 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:54.061477 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:54.965110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:54.965010 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:55.202572 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:55.202535 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:55.631462 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:55.631404 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:04:57.378165 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:04:57.377526 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:00.057525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:00.057299 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:00.272264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:00.272221 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:00.601480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:00.600398 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:00.847216 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:00.847174 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:00.885101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:00.884998 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:01.250352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:01.248922 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:01.272867 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:01.272787 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:05:01.289924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:01.273596 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:05:01.898925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:01.891650 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:01.898925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:01.893208 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:05:01.923356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:01.913121 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 17:05:01.923356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:01.914466 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:05:02.171516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:02.171416 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:02.171970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:02.171889 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:02.395813 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:02.394210 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:02.934328 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:02.934266 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:02.947913 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:02.935428 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:03.389022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:03.325085 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:03.398609 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:03.398564 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:03.956678 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:05:03.956638 4828 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.471s" Oct 30 17:05:03.966823 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:03.966767 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 17:05:04.084958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:04.067779 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-p7tp8" podUID="86d11598-b8b5-4653-93ab-438cf26e63eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 17:05:04.123402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:04.123360 4828 patch_prober.go:29] interesting pod/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:04.123706 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:04.123654 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" podUID="aab33acc-67cf-4cc4-b180-50cdee10b872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:04.233290 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:04.230810 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:05:04.233985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:04.233777 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:05:04.238737 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:04.238716 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:05:04.459436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:04.459395 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:06.187716 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:06.187680 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:07.806966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:07.783379 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:07.916202 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:05:07.916168 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:05:08.864943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:08.864865 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:09.014481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:09.014437 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:09.481943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:09.480788 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:09.656699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:09.656651 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:11.266257 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:11.266154 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:12.979812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:12.976762 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:13.175163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.175119 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:13.380073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.380035 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn"] Oct 30 17:05:13.398180 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.398140 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.416979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.416934 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7eda4908-openshift-default-dockercfg-fjtrw\"" Oct 30 17:05:13.431621 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.431587 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn"] Oct 30 17:05:13.523506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.508795 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r24lj\" (UniqueName: \"kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-kube-api-access-r24lj\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.576958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.566177 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-token\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.576958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.566278 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-podinfo\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.576958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.566358 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-credential-socket\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.576958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.566396 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-data\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.576958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.566426 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-envoy\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.576958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.566551 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-certs\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.576958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.566607 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-socket\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.576958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.566636 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istiod-ca-cert\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.608924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.600558 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.696671 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-certs\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.696742 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-socket\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.696772 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istiod-ca-cert\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.696929 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r24lj\" (UniqueName: \"kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-kube-api-access-r24lj\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.697020 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-token\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.697091 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-podinfo\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.697157 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-credential-socket\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.697189 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-data\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.697215 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-envoy\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.697365 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-socket\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.697411 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-certs\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.697652 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-credential-socket\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.698345 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istiod-ca-cert\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.700972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.698852 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-data\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.746109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.746075 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-envoy\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.778463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.778426 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:13.782788 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.782719 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-token\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.788982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.783605 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-podinfo\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:13.792325 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:13.792232 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r24lj\" (UniqueName: \"kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-kube-api-access-r24lj\") pod \"e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:14.041626 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:14.041585 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:14.386969 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:05:14.386893 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fa6a03_4ccd_4e0a_81de_9e406e1894a0.slice/crio-c8c526b9b6239f3611e1902baa7e807375d66693c3126702a9cf9c7784c80e56 WatchSource:0}: Error finding container c8c526b9b6239f3611e1902baa7e807375d66693c3126702a9cf9c7784c80e56: Status 404 returned error can't find the container with id c8c526b9b6239f3611e1902baa7e807375d66693c3126702a9cf9c7784c80e56 Oct 30 17:05:14.389892 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:14.389848 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:14.390001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:14.389956 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:14.390045 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:14.390031 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:14.390645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:14.390533 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn"] Oct 30 17:05:15.061596 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:15.061556 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:15.214478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:15.214398 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:15.343982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:15.336885 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" event={"ID":"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0","Type":"ContainerStarted","Data":"b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508"} Oct 30 17:05:15.343982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:15.336969 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" event={"ID":"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0","Type":"ContainerStarted","Data":"c8c526b9b6239f3611e1902baa7e807375d66693c3126702a9cf9c7784c80e56"} Oct 30 17:05:16.042654 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:16.042613 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:16.048441 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:16.048409 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:16.108749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:16.108690 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" podStartSLOduration=3.108672305 podStartE2EDuration="3.108672305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:15.425717026 +0000 UTC m=+1093.071227207" watchObservedRunningTime="2026-10-30 17:05:16.108672305 +0000 UTC m=+1093.754182485" Oct 30 17:05:16.135985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:16.135883 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:16.298970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:16.289279 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:16.411967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:16.398223 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:16.422970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:16.420394 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:05:16.466793 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:16.466745 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:16.930106 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:16.929813 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq"] Oct 30 17:05:16.943427 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:16.940385 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:16.967143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:16.967106 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-952bc16e-openshift-default-dockercfg-w85kz\"" Oct 30 17:05:16.977022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:16.976879 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq"] Oct 30 17:05:17.010401 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.010367 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-istio-token\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.010607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.010419 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqtq6\" (UniqueName: \"kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-kube-api-access-fqtq6\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.010607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.010452 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04832433-d5db-4505-83ca-24001a6b2350-istiod-ca-cert\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.010607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.010553 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-envoy\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.010607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.010587 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-credential-socket\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.010866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.010788 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-socket\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.010866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.010833 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04832433-d5db-4505-83ca-24001a6b2350-istio-podinfo\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.011000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.010938 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-certs\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.011058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.011029 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-data\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.031418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.031247 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:17.118928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.118125 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-credential-socket\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.118928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.118274 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-socket\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.118928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.118303 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04832433-d5db-4505-83ca-24001a6b2350-istio-podinfo\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.118928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.118367 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-certs\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.118928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.118569 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-data\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.118928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.118753 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-istio-token\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.118928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.118820 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqtq6\" (UniqueName: \"kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-kube-api-access-fqtq6\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.118928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.118855 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04832433-d5db-4505-83ca-24001a6b2350-istiod-ca-cert\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.118928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.118929 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-envoy\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.136925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.135018 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-certs\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.136925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.135279 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-credential-socket\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.136925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.135517 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-socket\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.141932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.139479 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04832433-d5db-4505-83ca-24001a6b2350-istio-podinfo\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.141932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.140069 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-data\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.141932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.141368 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04832433-d5db-4505-83ca-24001a6b2350-istiod-ca-cert\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.149935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.147374 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-envoy\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.350891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.350853 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqtq6\" (UniqueName: \"kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-kube-api-access-fqtq6\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.387595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.387559 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-istio-token\") pod \"e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.632562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.632235 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:17.798814 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:17.797250 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:18.018927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:18.018883 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:18.159653 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:05:18.159622 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04832433_d5db_4505_83ca_24001a6b2350.slice/crio-e480f1fb6e3549634b8d68b3b60008fb5ad89fdc396d0abad06f56b694e97565 WatchSource:0}: Error finding container e480f1fb6e3549634b8d68b3b60008fb5ad89fdc396d0abad06f56b694e97565: Status 404 returned error can't find the container with id e480f1fb6e3549634b8d68b3b60008fb5ad89fdc396d0abad06f56b694e97565 Oct 30 17:05:18.162751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:18.162711 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:18.162870 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:18.162797 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:18.162870 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:18.162827 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:18.163098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:18.163075 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq"] Oct 30 17:05:18.787999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:18.787948 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" event={"ID":"04832433-d5db-4505-83ca-24001a6b2350","Type":"ContainerStarted","Data":"cf5a4c2e9b7c0e7410a12fe3bcd1d9983bab44ffc4f29a73491cb6797e6c882e"} Oct 30 17:05:18.788567 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:18.788016 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" event={"ID":"04832433-d5db-4505-83ca-24001a6b2350","Type":"ContainerStarted","Data":"e480f1fb6e3549634b8d68b3b60008fb5ad89fdc396d0abad06f56b694e97565"} Oct 30 17:05:18.842144 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:18.842110 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:18.853320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:18.853271 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" podStartSLOduration=2.853257638 podStartE2EDuration="2.853257638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:18.849597674 +0000 UTC m=+1096.495107854" watchObservedRunningTime="2026-10-30 17:05:18.853257638 +0000 UTC m=+1096.498767818" Oct 30 17:05:19.021593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:19.021546 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:19.426199 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:19.426158 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:19.641007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:19.637078 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:19.666928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:19.666186 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:19.801556 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:19.801518 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:19.803136 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:19.803112 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:05:19.804594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:19.804570 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:19.965779 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:19.965663 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:20.121693 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:20.121583 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:20.697880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:20.697831 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:20.865615 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:20.865576 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:21.655651 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:21.655610 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:22.036415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:22.036375 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:22.189466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:22.189374 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:22.466602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:22.466502 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:22.624458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:22.624416 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:23.835648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:23.835560 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:25.558960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:25.558914 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:28.405086 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:28.405039 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:28.980010 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:28.943054 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 17:05:29.306081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:29.305965 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:05:29.428574 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:29.428520 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:29.430571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:29.430544 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:05:29.632099 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:29.608744 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:30.385926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:30.043794 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:30.385926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:30.104055 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:05:30.453707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:30.453651 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:30.588558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:30.588497 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:05:30.694925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:30.694813 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:05:30.705181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:30.699679 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:05:30.727093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:30.699718 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r" podUID="6ab66bcb-c480-4c94-abe5-307e17932f99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:30.938934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:30.938474 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:05:31.138593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:31.130321 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.436979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:31.435636 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:31.498947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:31.479713 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.498947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:31.480243 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.591350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:31.591302 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:05:31.617925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:31.599409 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:31.617925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:31.599486 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" podUID="7beb4126-0039-4932-be23-21d475782c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:31.742990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:31.725784 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:05:31.941491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:31.941115 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:31.941491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:31.941187 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" podUID="08423875-76fe-41f6-995a-6c5832230d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:32.042203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:32.041771 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:32.042203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:32.041859 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" podUID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:32.108920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:32.106055 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:32.108920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:32.106143 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" podUID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:32.205967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:32.199230 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:32.336050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:32.327093 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:32.336050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:32.327163 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" podUID="e88601bd-82a5-47c7-be65-67915627db91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:32.499248 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:32.499198 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:32.779482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:32.573080 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:05:32.903975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:32.903925 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:32.904210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:32.783674 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:32.904624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:32.597032 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:33.220104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:33.219989 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:33.765062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:33.765009 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:33.784798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:33.784764 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:05:33.871946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:33.862015 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:33.922937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:33.916652 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:05:33.926346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:33.924103 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:05:34.287580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:34.287464 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:05:34.290619 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:34.289797 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:05:34.943501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:34.943437 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:05:35.104139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:35.104107 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:05:35.816857 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:35.816820 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:35.977338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:35.977305 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:36.208810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:36.208707 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:36.865495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:36.865450 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:37.756034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:37.755994 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg"] Oct 30 17:05:37.760672 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:37.760652 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:37.772321 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:37.772287 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:37.809592 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:37.804966 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg"] Oct 30 17:05:37.899927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:37.880985 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:37.900285 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:37.900248 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:37.900497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:37.900471 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:37.900698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:37.900679 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:37.901005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:37.900879 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:37.901258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:37.901236 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:37.901556 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:37.901534 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbjkt\" (UniqueName: \"kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-kube-api-access-fbjkt\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:37.913960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:37.902116 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:37.927126 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:37.927056 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:37.985570 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:05:37.985406 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:05:38.029300 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.029264 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbjkt\" (UniqueName: \"kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-kube-api-access-fbjkt\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.029482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.029382 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.029482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.029410 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.029482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.029451 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.029482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.029475 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.029717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.029514 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.029717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.029565 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.029717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.029625 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.029717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.029711 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.029984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.029941 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-credential-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.030117 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.030093 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-socket\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.030212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.030183 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-certs\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.030274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.030212 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-data\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.030501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.030478 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istiod-ca-cert\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.032048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.032025 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-envoy\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.032262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.032242 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-podinfo\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.055442 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.055403 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbjkt\" (UniqueName: \"kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-kube-api-access-fbjkt\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.056220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.056195 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-token\") pod \"e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.078498 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.078461 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:38.216622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.216548 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx"] Oct 30 17:05:38.226440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.226412 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.241746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.241713 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx"] Oct 30 17:05:38.333917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.333819 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.333917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.333878 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stq5d\" (UniqueName: \"kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-kube-api-access-stq5d\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.334125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.333995 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.334125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.334043 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.334125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.334116 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.334276 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.334261 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.334322 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.334310 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.334399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.334385 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.334458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.334426 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.411379 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:05:38.411334 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a5a6a7_8d69_4f16_ba3f_8aebd74dd917.slice/crio-07d900adf172fd55f71771662dcb6c65899d5af2c347f1f46075b2e775a0fe47 WatchSource:0}: Error finding container 07d900adf172fd55f71771662dcb6c65899d5af2c347f1f46075b2e775a0fe47: Status 404 returned error can't find the container with id 07d900adf172fd55f71771662dcb6c65899d5af2c347f1f46075b2e775a0fe47 Oct 30 17:05:38.411940 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.411870 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg"] Oct 30 17:05:38.414722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.414682 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:38.414816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.414769 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:38.414861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.414813 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:38.435962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.435941 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.436041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.435982 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.436041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.436015 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.436150 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.436135 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.436211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.436179 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.436279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.436224 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.436341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.436274 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.436395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.436348 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.436454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.436395 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-data\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.436454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.436410 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stq5d\" (UniqueName: \"kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-kube-api-access-stq5d\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.436542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.436477 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-certs\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.436599 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.436574 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-credential-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.436680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.436663 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-socket\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.437069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.437037 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istiod-ca-cert\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.438762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.438737 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-envoy\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.438947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.438926 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-podinfo\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.460807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.460783 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-token\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.461028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.461002 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stq5d\" (UniqueName: \"kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-kube-api-access-stq5d\") pod \"e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.500747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.500710 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" event={"ID":"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917","Type":"ContainerStarted","Data":"07d900adf172fd55f71771662dcb6c65899d5af2c347f1f46075b2e775a0fe47"} Oct 30 17:05:38.549624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.549586 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:38.554932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.554855 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" podStartSLOduration=1.554839223 podStartE2EDuration="1.554839223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:38.551664209 +0000 UTC m=+1116.197174390" watchObservedRunningTime="2026-10-30 17:05:38.554839223 +0000 UTC m=+1116.200349405" Oct 30 17:05:38.894290 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:05:38.894258 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5baf94a_dc53_4365_ab04_2f7b0ad65bdf.slice/crio-e88b38e446d295188f52a6e0c834ba0223010b793cc7703838ab833f460653f7 WatchSource:0}: Error finding container e88b38e446d295188f52a6e0c834ba0223010b793cc7703838ab833f460653f7: Status 404 returned error can't find the container with id e88b38e446d295188f52a6e0c834ba0223010b793cc7703838ab833f460653f7 Oct 30 17:05:38.895335 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.895309 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx"] Oct 30 17:05:38.897892 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.897854 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:38.898040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.897996 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:38.898100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:38.898059 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:39.088332 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.088299 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:39.094378 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.094033 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:05:39.094378 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.094094 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" podUID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:05:39.430143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.430113 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd"] Oct 30 17:05:39.447342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.447304 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:39.458145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.458107 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd"] Oct 30 17:05:39.473007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.466946 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7eda4908-serving-cert\"" Oct 30 17:05:39.532000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.531960 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" event={"ID":"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf","Type":"ContainerStarted","Data":"a2ae1caf0c6265b3b08f76d4322571415178a0cc5860dcae9fac7e651cb8effb"} Oct 30 17:05:39.532259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.532238 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" event={"ID":"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf","Type":"ContainerStarted","Data":"e88b38e446d295188f52a6e0c834ba0223010b793cc7703838ab833f460653f7"} Oct 30 17:05:39.536236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.536207 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" event={"ID":"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917","Type":"ContainerStarted","Data":"559d68d9af1d6223213120b8083ff41c517d9f2f349d5d7f995534f834749ea8"} Oct 30 17:05:39.549829 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.549798 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:39.556842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.556810 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-maas-api-tls\") pod \"maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:39.557100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.557077 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-metrics-certs\") pod \"maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:39.557352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.557333 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmp8n\" (UniqueName: \"kubernetes.io/projected/34a3859d-12e6-4d30-a13e-6b54738eab22-kube-api-access-dmp8n\") pod \"maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:39.557709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.557690 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a3859d-12e6-4d30-a13e-6b54738eab22-openshift-service-ca\") pod \"maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:39.643924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.633635 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" podStartSLOduration=1.633610719 podStartE2EDuration="1.633610719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:39.629671938 +0000 UTC m=+1117.275182110" watchObservedRunningTime="2026-10-30 17:05:39.633610719 +0000 UTC m=+1117.279120900" Oct 30 17:05:39.648922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.647968 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:39.660688 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.660409 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a3859d-12e6-4d30-a13e-6b54738eab22-openshift-service-ca\") pod \"maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:39.660861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.660794 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-maas-api-tls\") pod \"maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:39.660861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.660839 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-metrics-certs\") pod \"maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:39.661024 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.661011 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmp8n\" (UniqueName: \"kubernetes.io/projected/34a3859d-12e6-4d30-a13e-6b54738eab22-kube-api-access-dmp8n\") pod \"maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:39.661194 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.661167 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a3859d-12e6-4d30-a13e-6b54738eab22-openshift-service-ca\") pod \"maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:39.663992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.663963 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-metrics-certs\") pod \"maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:39.665837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.665815 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-maas-api-tls\") pod \"maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:39.761097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.759633 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmp8n\" (UniqueName: \"kubernetes.io/projected/34a3859d-12e6-4d30-a13e-6b54738eab22-kube-api-access-dmp8n\") pod \"maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:39.815931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.813741 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:39.835552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:39.835272 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:40.097985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:40.081890 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:05:40.097985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:40.081975 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" podUID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:05:40.646913 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:40.646090 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 17:05:40.646913 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:40.646152 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" podUID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 17:05:40.663307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:40.663274 4828 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:05:40.726939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:40.725117 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd"] Oct 30 17:05:40.732255 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:40.731523 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" event={"ID":"34a3859d-12e6-4d30-a13e-6b54738eab22","Type":"ContainerStarted","Data":"3c53153c16c4c7d4d538c50a3b4764aeb2c06053661dbd9a80d066220532c656"} Oct 30 17:05:41.085921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:41.080367 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:05:41.085921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:41.080435 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" podUID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:05:41.550965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:41.550877 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 17:05:41.550965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:41.550958 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" podUID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 17:05:41.744048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:41.743048 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" event={"ID":"34a3859d-12e6-4d30-a13e-6b54738eab22","Type":"ContainerStarted","Data":"74226fb73f023bc5aacf04e67a79d0d0866771834cbd3cffe724f9ee13acc56c"} Oct 30 17:05:41.744048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:41.743978 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:41.929778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:41.929740 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:42.087925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:42.084309 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:05:42.087925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:42.084392 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" podUID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:05:42.113925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:42.110498 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:42.346941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:42.346818 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:42.550117 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:42.550063 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 17:05:42.550431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:42.550125 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" podUID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 17:05:42.843922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:42.840055 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:43.076931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:43.075861 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:43.088970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:43.085121 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:05:43.088970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:43.085183 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" podUID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:05:43.590925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:43.572123 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 17:05:43.590925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:43.572181 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" podUID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 17:05:44.084922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.084760 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:05:44.084922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.084827 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" podUID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:05:44.437391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.437262 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" podStartSLOduration=4.916571677 podStartE2EDuration="5.437240102s" totalImagesPullingTime="520.668425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:41.802321718 +0000 UTC m=+1119.447831898" watchObservedRunningTime="2026-10-30 17:05:44.437240102 +0000 UTC m=+1122.082750285" Oct 30 17:05:44.440630 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.440600 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26"] Oct 30 17:05:44.447203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.447180 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:44.447614 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.447587 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26"] Oct 30 17:05:44.537441 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.537409 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-952bc16e-serving-cert\"" Oct 30 17:05:44.550773 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.550740 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 17:05:44.551047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.551018 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" podUID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 17:05:44.552687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.552660 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-maas-api-tls\") pod \"maas-api-e2e-ait-952bc16e-5f55977df9-2ct26\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:44.552946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.552926 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9467d3b9-f7da-43e8-a841-e20944fbb3fe-openshift-service-ca\") pod \"maas-api-e2e-ait-952bc16e-5f55977df9-2ct26\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:44.553545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.553523 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcjwv\" (UniqueName: \"kubernetes.io/projected/9467d3b9-f7da-43e8-a841-e20944fbb3fe-kube-api-access-fcjwv\") pod \"maas-api-e2e-ait-952bc16e-5f55977df9-2ct26\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:44.553818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.553798 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-metrics-certs\") pod \"maas-api-e2e-ait-952bc16e-5f55977df9-2ct26\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:44.554746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.554575 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:44.766980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.750442 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:44.918771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.918735 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-maas-api-tls\") pod \"maas-api-e2e-ait-952bc16e-5f55977df9-2ct26\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:44.919104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.919077 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9467d3b9-f7da-43e8-a841-e20944fbb3fe-openshift-service-ca\") pod \"maas-api-e2e-ait-952bc16e-5f55977df9-2ct26\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:44.919360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.919336 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcjwv\" (UniqueName: \"kubernetes.io/projected/9467d3b9-f7da-43e8-a841-e20944fbb3fe-kube-api-access-fcjwv\") pod \"maas-api-e2e-ait-952bc16e-5f55977df9-2ct26\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:44.919626 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.919609 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-metrics-certs\") pod \"maas-api-e2e-ait-952bc16e-5f55977df9-2ct26\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:44.922410 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:44.921739 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9467d3b9-f7da-43e8-a841-e20944fbb3fe-openshift-service-ca\") pod \"maas-api-e2e-ait-952bc16e-5f55977df9-2ct26\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:45.023977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:45.016595 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-metrics-certs\") pod \"maas-api-e2e-ait-952bc16e-5f55977df9-2ct26\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:45.023977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:45.020447 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcjwv\" (UniqueName: \"kubernetes.io/projected/9467d3b9-f7da-43e8-a841-e20944fbb3fe-kube-api-access-fcjwv\") pod \"maas-api-e2e-ait-952bc16e-5f55977df9-2ct26\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:45.044741 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:45.044502 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-maas-api-tls\") pod \"maas-api-e2e-ait-952bc16e-5f55977df9-2ct26\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:45.088763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:45.088311 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:05:45.088763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:45.088375 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" podUID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:05:45.135921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:45.135653 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:45.606981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:45.556308 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 17:05:45.606981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:45.556372 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" podUID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 17:05:46.009294 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:46.009258 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:46.043817 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:46.035488 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26"] Oct 30 17:05:46.044606 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:05:46.044567 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9467d3b9_f7da_43e8_a841_e20944fbb3fe.slice/crio-691453141e5496851d4f8078dccd234fec395511982b8c698ba6c1b97b012f7a WatchSource:0}: Error finding container 691453141e5496851d4f8078dccd234fec395511982b8c698ba6c1b97b012f7a: Status 404 returned error can't find the container with id 691453141e5496851d4f8078dccd234fec395511982b8c698ba6c1b97b012f7a Oct 30 17:05:46.103094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:46.103049 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 17:05:46.103752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:46.103715 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" podUID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 17:05:46.192933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:46.191138 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:46.558923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:46.551769 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 17:05:46.558923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:46.551831 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" podUID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 17:05:46.885217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:46.873623 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:47.064802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:47.064743 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" event={"ID":"9467d3b9-f7da-43e8-a841-e20944fbb3fe","Type":"ContainerStarted","Data":"d2673cb41f5bd466b85de243c4a9c0629a5452e09876cd17efb5df12523ce9fc"} Oct 30 17:05:47.065013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:47.064835 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" event={"ID":"9467d3b9-f7da-43e8-a841-e20944fbb3fe","Type":"ContainerStarted","Data":"691453141e5496851d4f8078dccd234fec395511982b8c698ba6c1b97b012f7a"} Oct 30 17:05:47.065013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:47.064871 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:47.116823 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:47.116762 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" podStartSLOduration=2.658589618 podStartE2EDuration="3.116741961s" totalImagesPullingTime="458.152343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:47.110125379 +0000 UTC m=+1124.755635574" watchObservedRunningTime="2026-10-30 17:05:47.116741961 +0000 UTC m=+1124.762252142" Oct 30 17:05:47.215366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:47.215260 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:47.216643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:47.216527 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:47.217470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:47.217418 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:05:47.350434 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:47.350381 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:47.509174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:47.509139 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:47.551106 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:47.551072 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 17:05:47.551273 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:47.551136 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" podUID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 17:05:47.864988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:47.861183 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:05:48.634385 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:48.634350 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:48.634993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:48.634967 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:48.635833 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:48.635809 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:05:49.423728 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:49.423693 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:51.279783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:51.279730 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:52.838140 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:52.838095 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:53.110603 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.110529 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:05:53.123014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.122982 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:53.482625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.482537 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2"] Oct 30 17:05:53.488714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.488685 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.523418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.523368 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2"] Oct 30 17:05:53.546432 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.546370 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.546640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.546445 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.546640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.546549 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.546762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.546685 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.546762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.546740 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.546841 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.546766 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.547322 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.547259 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chzms\" (UniqueName: \"kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-kube-api-access-chzms\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.547831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.547387 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.547831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.547547 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.649315 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.649276 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.649487 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.649365 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.649487 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.649474 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.649569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.649539 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.649602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.649565 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.649934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.649742 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chzms\" (UniqueName: \"kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-kube-api-access-chzms\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.649934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.649757 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.649934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.649824 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.650146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.649944 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.650146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.649985 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.650146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.650058 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.650320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.650188 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.650320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.650236 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.650407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.650333 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.652432 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.652412 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.652548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.652526 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.674599 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.674565 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.674762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.674657 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chzms\" (UniqueName: \"kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-kube-api-access-chzms\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:53.806931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:53.806879 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:54.084991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:54.084881 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:54.135181 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:05:54.135145 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703615b6_f313_4e6d_a35c_6d9caf3d3ae9.slice/crio-07a9426b3c5be9ea213c7da7b1d1be0a5a7d5fd3d5d362923ae7548e593e6f4a WatchSource:0}: Error finding container 07a9426b3c5be9ea213c7da7b1d1be0a5a7d5fd3d5d362923ae7548e593e6f4a: Status 404 returned error can't find the container with id 07a9426b3c5be9ea213c7da7b1d1be0a5a7d5fd3d5d362923ae7548e593e6f4a Oct 30 17:05:54.137121 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:54.137098 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2"] Oct 30 17:05:54.138145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:54.138113 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:54.138226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:54.138175 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:54.138226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:54.138205 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:05:54.172148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:54.172099 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" event={"ID":"703615b6-f313-4e6d-a35c-6d9caf3d3ae9","Type":"ContainerStarted","Data":"07a9426b3c5be9ea213c7da7b1d1be0a5a7d5fd3d5d362923ae7548e593e6f4a"} Oct 30 17:05:54.533940 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:54.530798 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:55.077157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:55.077123 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:55.184435 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:55.184384 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" event={"ID":"703615b6-f313-4e6d-a35c-6d9caf3d3ae9","Type":"ContainerStarted","Data":"6d04aa9a1a97dacd004f2d54dde6a4a54e3e66900768a95a23962043beb493bb"} Oct 30 17:05:55.250953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:55.250878 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" podStartSLOduration=2.250861312 podStartE2EDuration="2.250861312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:55.245305733 +0000 UTC m=+1132.890815914" watchObservedRunningTime="2026-10-30 17:05:55.250861312 +0000 UTC m=+1132.896371493" Oct 30 17:05:55.323955 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:55.323920 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:55.807801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:55.807750 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:05:55.809478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:55.809441 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 17:05:55.809633 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:55.809506 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 17:05:56.531925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:56.531186 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:56.808270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:56.808181 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 17:05:56.808270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:56.808237 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 17:05:57.807448 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:57.807410 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 17:05:57.808015 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:57.807492 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 17:05:58.287923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:58.287437 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:05:58.807482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:58.807433 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 17:05:58.808037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:58.807498 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 17:05:59.807556 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:59.807515 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 17:05:59.808105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:59.807578 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 17:05:59.842518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:05:59.842479 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:00.353921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:00.351632 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:00.810322 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:00.810265 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 17:06:00.810978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:00.810875 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 17:06:01.809924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:01.807327 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 17:06:01.809924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:01.807396 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 17:06:02.449496 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:02.449452 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:03.626982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:03.622994 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:03.741176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:03.741113 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:03.841318 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:03.841022 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:03.884189 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:03.884065 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:06:03.884491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:03.884442 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:04.229164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:04.229058 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:04.497724 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:04.497046 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:04.681534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:04.681481 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:06:04.889925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:04.741791 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:04.889925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:04.889746 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:06:04.889925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:04.889806 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:06:04.917934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:04.916624 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:06:04.917934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:04.916690 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:05.251992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:05.240974 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:05.485396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:05.485362 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:06:05.521975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:05.521050 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:05.529929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:05.522649 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:06:06.159107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:06.159035 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:06:06.198939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:06.164360 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:06:06.585022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:06.295219 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:06.665980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:06.173301 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:06:06.665980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:06.659834 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:06.665980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:06.589919 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:06.665980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:06.594511 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 30 17:06:06.665980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:06.534023 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:06:06.665980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:06.541072 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:06:06.665980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:06.659973 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:06.721702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:06.721663 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:06.896930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:06.896828 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:06:07.333025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:07.332182 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:06:07.780991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:07.780009 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:07.794605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:07.794560 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 17:06:07.847950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:07.832558 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:06:07.847950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:07.832620 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:07.902952 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:07.893522 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:08.080973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:08.073810 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:08.185924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:08.160150 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:08.315929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:08.312599 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:06:08.339126 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:08.339029 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:08.384938 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:06:08.384872 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:06:08.414952 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:08.396582 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:08.798869 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:08.798739 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:06:08.813327 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:08.813294 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:06:08.862368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:08.862339 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:06:08.874931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:08.868152 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:06:08.968016 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:08.959856 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:06:09.060705 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:09.060548 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:09.069054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:09.069025 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:06:11.310929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:11.300007 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:13.424967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:13.424859 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:14.770745 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:14.770706 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:14.926789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:14.925670 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:15.769926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:15.767962 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:17.431090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:17.431046 4828 patch_prober.go:29] interesting pod/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:06:17.431503 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:17.431116 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:17.586678 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:17.586517 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:19.211931 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:06:19.211876 4828 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fa6a03_4ccd_4e0a_81de_9e406e1894a0.slice/crio-conmon-b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:06:19.219525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.219493 4828 generic.go:356] "Generic (PLEG): container finished" podID="04832433-d5db-4505-83ca-24001a6b2350" containerID="cf5a4c2e9b7c0e7410a12fe3bcd1d9983bab44ffc4f29a73491cb6797e6c882e" exitCode=0 Oct 30 17:06:19.219706 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.219560 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" event={"ID":"04832433-d5db-4505-83ca-24001a6b2350","Type":"ContainerDied","Data":"cf5a4c2e9b7c0e7410a12fe3bcd1d9983bab44ffc4f29a73491cb6797e6c882e"} Oct 30 17:06:19.220116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.220096 4828 scope.go:122] "RemoveContainer" containerID="cf5a4c2e9b7c0e7410a12fe3bcd1d9983bab44ffc4f29a73491cb6797e6c882e" Oct 30 17:06:19.221956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.221927 4828 generic.go:356] "Generic (PLEG): container finished" podID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerID="b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508" exitCode=0 Oct 30 17:06:19.222076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.222047 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:19.222137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.222082 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" event={"ID":"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0","Type":"ContainerDied","Data":"b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508"} Oct 30 17:06:19.222137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.222116 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:19.222246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.222164 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:19.222449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.222432 4828 scope.go:122] "RemoveContainer" containerID="b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508" Oct 30 17:06:19.223872 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.223846 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:19.224008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.223927 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:19.224008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.223966 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:19.712657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.708657 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:19.801925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:19.801517 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:06:20.255671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:20.255635 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" event={"ID":"04832433-d5db-4505-83ca-24001a6b2350","Type":"ContainerStarted","Data":"c993f833c41415a74e826ae39d886b670cd693827180f0c3cee2d772280153ad"} Oct 30 17:06:20.258786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:20.258759 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" event={"ID":"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0","Type":"ContainerStarted","Data":"8890b723f62f356a50ac52dc584942286e61483bca974a2b5c0268088a9ce2a9"} Oct 30 17:06:20.634097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:20.634054 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:06:20.634745 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:20.634713 4828 patch_prober.go:29] interesting pod/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 17:06:20.634865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:20.634772 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" podUID="04832433-d5db-4505-83ca-24001a6b2350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 17:06:21.042862 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:21.042826 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:06:21.043081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:21.043048 4828 patch_prober.go:29] interesting pod/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 17:06:21.043134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:21.043092 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 17:06:21.566715 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:21.566602 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:21.660072 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:21.647246 4828 patch_prober.go:29] interesting pod/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 17:06:21.660072 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:21.647309 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" podUID="04832433-d5db-4505-83ca-24001a6b2350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 17:06:22.042486 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:22.042448 4828 patch_prober.go:29] interesting pod/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 17:06:22.042706 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:22.042511 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 17:06:22.634006 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:22.633971 4828 patch_prober.go:29] interesting pod/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 17:06:22.634414 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:22.634029 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" podUID="04832433-d5db-4505-83ca-24001a6b2350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 17:06:23.042343 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.042301 4828 patch_prober.go:29] interesting pod/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 17:06:23.042549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.042369 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 17:06:23.568272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.568235 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb"] Oct 30 17:06:23.599872 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.599834 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.731137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.674017 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb"] Oct 30 17:06:23.731137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.675294 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.731137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.675379 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.731137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.675438 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008aa481-f9ff-4742-b198-d8556de19b3a-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.731137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.675539 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhls\" (UniqueName: \"kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-kube-api-access-vlhls\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.731137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.675581 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.731137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.675616 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008aa481-f9ff-4742-b198-d8556de19b3a-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.731137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.675749 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.731137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.675822 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.731137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.675876 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.731137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.690309 4828 patch_prober.go:29] interesting pod/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 17:06:23.731137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.690366 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" podUID="04832433-d5db-4505-83ca-24001a6b2350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 17:06:23.766390 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.766192 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:23.794854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.793028 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.794854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.793121 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.794854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.793174 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.794854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.793328 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.794854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.793380 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.794854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.793425 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008aa481-f9ff-4742-b198-d8556de19b3a-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.794854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.793526 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhls\" (UniqueName: \"kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-kube-api-access-vlhls\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.794854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.793557 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.794854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.793582 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008aa481-f9ff-4742-b198-d8556de19b3a-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.842038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.830708 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-certs\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.842038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.831021 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.842038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.831274 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-credential-socket\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.842038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.831787 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-data\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.863531 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.863489 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008aa481-f9ff-4742-b198-d8556de19b3a-istiod-ca-cert\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.948934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.948284 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-envoy\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.948934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.948689 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-istio-token\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.949331 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.949271 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008aa481-f9ff-4742-b198-d8556de19b3a-istio-podinfo\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:23.950366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:23.950334 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhls\" (UniqueName: \"kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-kube-api-access-vlhls\") pod \"e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:24.043812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:24.043777 4828 patch_prober.go:29] interesting pod/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 17:06:24.044030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:24.043828 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 17:06:24.223514 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:24.223391 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:24.633953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:24.633128 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb"] Oct 30 17:06:24.635486 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:24.635454 4828 patch_prober.go:29] interesting pod/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 17:06:24.635581 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:24.635530 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" podUID="04832433-d5db-4505-83ca-24001a6b2350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 17:06:24.643121 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:24.641841 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:24.643121 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:24.641957 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:24.643121 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:24.642007 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:25.042586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:25.042537 4828 patch_prober.go:29] interesting pod/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 17:06:25.043027 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:25.042675 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 17:06:25.366601 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:25.366560 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" event={"ID":"008aa481-f9ff-4742-b198-d8556de19b3a","Type":"ContainerStarted","Data":"1c0721e85d29228503a3b3edb628e911bdd74667d78955875fa9dc675b8f20a3"} Oct 30 17:06:25.366783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:25.366650 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" event={"ID":"008aa481-f9ff-4742-b198-d8556de19b3a","Type":"ContainerStarted","Data":"22e631e7f640d121b881f018f3b22a525b0bd0f5bfe53fd9e02c214c968551a6"} Oct 30 17:06:25.441130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:25.440949 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" podStartSLOduration=2.440931192 podStartE2EDuration="2.440931192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:25.440001122 +0000 UTC m=+1163.085511305" watchObservedRunningTime="2026-10-30 17:06:25.440931192 +0000 UTC m=+1163.086441373" Oct 30 17:06:25.591950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:25.591892 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:25.634022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:25.633935 4828 patch_prober.go:29] interesting pod/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 17:06:25.634356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:25.634324 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" podUID="04832433-d5db-4505-83ca-24001a6b2350" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 17:06:26.042802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:26.042756 4828 patch_prober.go:29] interesting pod/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 30 17:06:26.043328 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:26.042844 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 30 17:06:26.225291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:26.225254 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:26.226099 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:26.226068 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 17:06:26.226284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:26.226256 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 17:06:26.637729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:26.637690 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:06:26.638172 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:26.638150 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:06:26.638883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:26.638865 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:06:27.050008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:27.049981 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:06:27.051136 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:27.051016 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:06:27.052077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:27.052053 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:06:27.055954 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:27.055911 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd"] Oct 30 17:06:27.056166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:27.056142 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" podUID="34a3859d-12e6-4d30-a13e-6b54738eab22" containerName="maas-api" containerID="cri-o://74226fb73f023bc5aacf04e67a79d0d0866771834cbd3cffe724f9ee13acc56c" gracePeriod=30 Oct 30 17:06:27.150352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:27.150313 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:27.224757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:27.224342 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 17:06:27.224757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:27.224409 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 17:06:28.071938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.070716 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:28.120936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.117569 4828 generic.go:356] "Generic (PLEG): container finished" podID="34a3859d-12e6-4d30-a13e-6b54738eab22" containerID="74226fb73f023bc5aacf04e67a79d0d0866771834cbd3cffe724f9ee13acc56c" exitCode=0 Oct 30 17:06:28.120936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.118778 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" event={"ID":"34a3859d-12e6-4d30-a13e-6b54738eab22","Type":"ContainerDied","Data":"74226fb73f023bc5aacf04e67a79d0d0866771834cbd3cffe724f9ee13acc56c"} Oct 30 17:06:28.232936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.226325 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 17:06:28.232936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.226383 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 17:06:28.233388 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.233362 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:06:28.340980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.330088 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a3859d-12e6-4d30-a13e-6b54738eab22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a3859d-12e6-4d30-a13e-6b54738eab22-openshift-service-ca\") pod \"34a3859d-12e6-4d30-a13e-6b54738eab22\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " Oct 30 17:06:28.341334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.341298 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-maas-api-tls\") pod \"34a3859d-12e6-4d30-a13e-6b54738eab22\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " Oct 30 17:06:28.341759 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.341736 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-metrics-certs\") pod \"34a3859d-12e6-4d30-a13e-6b54738eab22\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " Oct 30 17:06:28.342025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.341996 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a3859d-12e6-4d30-a13e-6b54738eab22-kube-api-access-dmp8n\" (UniqueName: \"kubernetes.io/projected/34a3859d-12e6-4d30-a13e-6b54738eab22-kube-api-access-dmp8n\") pod \"34a3859d-12e6-4d30-a13e-6b54738eab22\" (UID: \"34a3859d-12e6-4d30-a13e-6b54738eab22\") " Oct 30 17:06:28.350946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.349297 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:28.462302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.334331 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a3859d-12e6-4d30-a13e-6b54738eab22-openshift-service-ca" pod "34a3859d-12e6-4d30-a13e-6b54738eab22" (UID: "34a3859d-12e6-4d30-a13e-6b54738eab22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:28.462524 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.367358 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a3859d-12e6-4d30-a13e-6b54738eab22-kube-api-access-dmp8n" pod "34a3859d-12e6-4d30-a13e-6b54738eab22" (UID: "34a3859d-12e6-4d30-a13e-6b54738eab22"). InnerVolumeSpecName "kube-api-access-dmp8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:28.462609 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.391205 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-metrics-certs" pod "34a3859d-12e6-4d30-a13e-6b54738eab22" (UID: "34a3859d-12e6-4d30-a13e-6b54738eab22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:28.462678 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.451080 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-maas-api-tls" pod "34a3859d-12e6-4d30-a13e-6b54738eab22" (UID: "34a3859d-12e6-4d30-a13e-6b54738eab22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:28.551147 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.551114 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:28.551147 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.551149 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmp8n\" (UniqueName: \"kubernetes.io/projected/34a3859d-12e6-4d30-a13e-6b54738eab22-kube-api-access-dmp8n\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:28.551409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.551191 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a3859d-12e6-4d30-a13e-6b54738eab22-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:28.551409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.551390 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34a3859d-12e6-4d30-a13e-6b54738eab22-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:28.912618 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:28.912579 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:29.140594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:29.140472 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:29.248039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:29.246657 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" Oct 30 17:06:29.262127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:29.257308 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" event={"ID":"34a3859d-12e6-4d30-a13e-6b54738eab22","Type":"ContainerDied","Data":"3c53153c16c4c7d4d538c50a3b4764aeb2c06053661dbd9a80d066220532c656"} Oct 30 17:06:29.262127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:29.257429 4828 scope.go:122] "RemoveContainer" containerID="74226fb73f023bc5aacf04e67a79d0d0866771834cbd3cffe724f9ee13acc56c" Oct 30 17:06:29.360936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:29.354486 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 17:06:29.361260 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:29.361221 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 17:06:29.424002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:29.410205 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:29.536231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:29.536199 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd"] Oct 30 17:06:29.545543 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:29.545515 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd"] Oct 30 17:06:30.065949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:30.048073 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn"] Oct 30 17:06:30.103923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:30.101385 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:30.332336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:30.332243 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 17:06:30.340030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:30.339985 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 17:06:30.616126 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:30.600530 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" containerID="cri-o://8890b723f62f356a50ac52dc584942286e61483bca974a2b5c0268088a9ce2a9" gracePeriod=30 Oct 30 17:06:30.697084 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:30.697048 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a3859d-12e6-4d30-a13e-6b54738eab22" path="/var/lib/kubelet/pods/34a3859d-12e6-4d30-a13e-6b54738eab22/volumes" Oct 30 17:06:30.774947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:30.771644 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7eda4908-54b486b4b4-rwjrd" podUID="34a3859d-12e6-4d30-a13e-6b54738eab22" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.104:8443/health\": context deadline exceeded" Oct 30 17:06:30.804416 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:30.803666 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:30.804416 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:30.803797 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:31.132473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.132436 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc"] Oct 30 17:06:31.133214 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.133185 4828 cpu_manager.go:414] "containerMap: removing container" podUID="34a3859d-12e6-4d30-a13e-6b54738eab22" containerName="maas-api" Oct 30 17:06:31.133214 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.133200 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a3859d-12e6-4d30-a13e-6b54738eab22" containerName="maas-api" Oct 30 17:06:31.133428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.133388 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a3859d-12e6-4d30-a13e-6b54738eab22" containerName="maas-api" Oct 30 17:06:31.138640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.138614 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.164362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.164326 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc"] Oct 30 17:06:31.183922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.180152 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-4aba8b6b-openshift-default-dockercfg-nkjcn\"" Oct 30 17:06:31.225516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.225467 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 17:06:31.225724 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.225524 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 17:06:31.226508 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.226473 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:31.260418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.260380 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-token\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.260590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.260432 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-credential-socket\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.260590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.260576 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-data\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.260671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.260620 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-certs\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.260834 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.260817 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-envoy\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.260957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.260938 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-socket\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.261019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.260970 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-podinfo\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.261118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.261101 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zw79\" (UniqueName: \"kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-kube-api-access-5zw79\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.261178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.261165 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d021b80-bf6b-4392-a3e9-465536ff5fae-istiod-ca-cert\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.367924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.367018 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-data\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.367924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.367088 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-certs\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.367924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.367249 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-envoy\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.367924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.367340 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-socket\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.367924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.367369 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-podinfo\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.367924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.367433 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zw79\" (UniqueName: \"kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-kube-api-access-5zw79\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.367924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.367512 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d021b80-bf6b-4392-a3e9-465536ff5fae-istiod-ca-cert\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.367924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.367577 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-token\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.367924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.367603 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-credential-socket\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.368844 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.368159 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-credential-socket\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.368844 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.368425 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-data\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.371250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.369187 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-certs\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.371250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.370482 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-socket\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.384026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.381564 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d021b80-bf6b-4392-a3e9-465536ff5fae-istiod-ca-cert\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.408930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.400651 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-podinfo\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.408930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.406240 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-envoy\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.408930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.408439 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:31.425944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.418640 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-token\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.425944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.419754 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zw79\" (UniqueName: \"kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-kube-api-access-5zw79\") pod \"e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.466604 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.466281 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:31.819101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:31.819062 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:32.226650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:32.226435 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 17:06:32.226650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:32.226489 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 17:06:32.432999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:32.421348 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc"] Oct 30 17:06:32.445879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:32.445829 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:32.446013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:32.445944 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:32.446053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:32.446011 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:32.751930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:32.751822 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" event={"ID":"6d021b80-bf6b-4392-a3e9-465536ff5fae","Type":"ContainerStarted","Data":"6be4b48064805f1165b55aae1abbfe1b71d88d6a6dd45569ba9f94b2c3edefe0"} Oct 30 17:06:32.752154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:32.751971 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" event={"ID":"6d021b80-bf6b-4392-a3e9-465536ff5fae","Type":"ContainerStarted","Data":"80279e807a3d54f3111669070a28a502aa9cd1b62dd85d8f566faa03b70336b5"} Oct 30 17:06:32.801920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:32.801171 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" podStartSLOduration=1.8011548080000002 podStartE2EDuration="1.801154808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:32.799153811 +0000 UTC m=+1170.444663993" watchObservedRunningTime="2026-10-30 17:06:32.801154808 +0000 UTC m=+1170.446664988" Oct 30 17:06:33.224342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:33.224302 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 17:06:33.224511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:33.224373 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 17:06:33.380450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:33.380409 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26"] Oct 30 17:06:33.382819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:33.380859 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" podUID="9467d3b9-f7da-43e8-a841-e20944fbb3fe" containerName="maas-api" containerID="cri-o://d2673cb41f5bd466b85de243c4a9c0629a5452e09876cd17efb5df12523ce9fc" gracePeriod=30 Oct 30 17:06:33.467978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:33.467945 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:33.473854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:33.473825 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:33.484138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:33.484037 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:34.169304 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.168689 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:34.228133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.224823 4828 generic.go:356] "Generic (PLEG): container finished" podID="9467d3b9-f7da-43e8-a841-e20944fbb3fe" containerID="d2673cb41f5bd466b85de243c4a9c0629a5452e09876cd17efb5df12523ce9fc" exitCode=0 Oct 30 17:06:34.240317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.240276 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" event={"ID":"9467d3b9-f7da-43e8-a841-e20944fbb3fe","Type":"ContainerDied","Data":"d2673cb41f5bd466b85de243c4a9c0629a5452e09876cd17efb5df12523ce9fc"} Oct 30 17:06:34.264210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.260368 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:34.264773 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.264738 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 17:06:34.265040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.265014 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 17:06:34.266049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.266029 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:06:34.271214 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.271186 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:06:34.386985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.386579 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-metrics-certs\") pod \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " Oct 30 17:06:34.386985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.386687 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-maas-api-tls\") pod \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " Oct 30 17:06:34.386985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.386750 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9467d3b9-f7da-43e8-a841-e20944fbb3fe-kube-api-access-fcjwv\" (UniqueName: \"kubernetes.io/projected/9467d3b9-f7da-43e8-a841-e20944fbb3fe-kube-api-access-fcjwv\") pod \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " Oct 30 17:06:34.386985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.386861 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9467d3b9-f7da-43e8-a841-e20944fbb3fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9467d3b9-f7da-43e8-a841-e20944fbb3fe-openshift-service-ca\") pod \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\" (UID: \"9467d3b9-f7da-43e8-a841-e20944fbb3fe\") " Oct 30 17:06:34.389135 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.389091 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9467d3b9-f7da-43e8-a841-e20944fbb3fe-openshift-service-ca" pod "9467d3b9-f7da-43e8-a841-e20944fbb3fe" (UID: "9467d3b9-f7da-43e8-a841-e20944fbb3fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:34.420225 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.420161 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-metrics-certs" pod "9467d3b9-f7da-43e8-a841-e20944fbb3fe" (UID: "9467d3b9-f7da-43e8-a841-e20944fbb3fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:34.420671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.420624 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9467d3b9-f7da-43e8-a841-e20944fbb3fe-kube-api-access-fcjwv" pod "9467d3b9-f7da-43e8-a841-e20944fbb3fe" (UID: "9467d3b9-f7da-43e8-a841-e20944fbb3fe"). InnerVolumeSpecName "kube-api-access-fcjwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:34.420963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.420871 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-maas-api-tls" pod "9467d3b9-f7da-43e8-a841-e20944fbb3fe" (UID: "9467d3b9-f7da-43e8-a841-e20944fbb3fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:34.506609 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.506557 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:34.507870 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.507850 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9467d3b9-f7da-43e8-a841-e20944fbb3fe-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:34.508552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.508532 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcjwv\" (UniqueName: \"kubernetes.io/projected/9467d3b9-f7da-43e8-a841-e20944fbb3fe-kube-api-access-fcjwv\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:34.508687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.508671 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9467d3b9-f7da-43e8-a841-e20944fbb3fe-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:34.619025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.618927 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:34.924022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.923980 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:34.932785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.932749 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq"] Oct 30 17:06:34.933627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:34.933564 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" podUID="04832433-d5db-4505-83ca-24001a6b2350" containerName="istio-proxy" containerID="cri-o://c993f833c41415a74e826ae39d886b670cd693827180f0c3cee2d772280153ad" gracePeriod=30 Oct 30 17:06:35.100881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:35.100846 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:35.341936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:35.340477 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" Oct 30 17:06:35.409939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:35.387890 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26" event={"ID":"9467d3b9-f7da-43e8-a841-e20944fbb3fe","Type":"ContainerDied","Data":"691453141e5496851d4f8078dccd234fec395511982b8c698ba6c1b97b012f7a"} Oct 30 17:06:35.409939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:35.388197 4828 scope.go:122] "RemoveContainer" containerID="d2673cb41f5bd466b85de243c4a9c0629a5452e09876cd17efb5df12523ce9fc" Oct 30 17:06:35.692650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:35.692614 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26"] Oct 30 17:06:35.725213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:35.725180 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-952bc16e-5f55977df9-2ct26"] Oct 30 17:06:35.861387 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:35.860875 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:35.863327 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:35.863306 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:35.867298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:35.867243 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:06:35.958547 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:35.958514 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:35.964746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:35.964722 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:06:36.051303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.051258 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-data\") pod \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " Oct 30 17:06:36.051492 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.051325 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-envoy\") pod \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " Oct 30 17:06:36.051492 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.051436 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-credential-socket\") pod \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " Oct 30 17:06:36.051620 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.051529 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-socket\") pod \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " Oct 30 17:06:36.051620 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.051601 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-certs\") pod \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " Oct 30 17:06:36.051763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.051712 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-data" pod "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" (UID: "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:36.051830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.051762 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istiod-ca-cert\") pod \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " Oct 30 17:06:36.051830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.051791 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-podinfo\") pod \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " Oct 30 17:06:36.051949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.051894 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-token\" (UniqueName: \"kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-token\") pod \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " Oct 30 17:06:36.052055 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.051953 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-kube-api-access-r24lj\" (UniqueName: \"kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-kube-api-access-r24lj\") pod \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\" (UID: \"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0\") " Oct 30 17:06:36.052122 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.052053 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-socket" pod "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" (UID: "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:36.052785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.052692 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istiod-ca-cert" pod "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" (UID: "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:36.052785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.052726 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:36.052785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.052751 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:36.052785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.052765 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-certs" pod "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" (UID: "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:36.053329 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.053310 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-credential-socket" pod "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" (UID: "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:36.054997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.054975 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-kube-api-access-r24lj" pod "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" (UID: "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0"). InnerVolumeSpecName "kube-api-access-r24lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:36.055207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.055182 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-podinfo" pod "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" (UID: "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:06:36.055333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.055299 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-envoy" pod "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" (UID: "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:36.058272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.058243 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-token" pod "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" (UID: "a2fa6a03-4ccd-4e0a-81de-9e406e1894a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:36.116070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.116034 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:36.154075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.154042 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:36.154075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.154073 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:36.154281 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.154087 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:36.154281 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.154096 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:36.154281 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.154105 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:36.154281 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.154113 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r24lj\" (UniqueName: \"kubernetes.io/projected/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-kube-api-access-r24lj\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:36.154281 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.154123 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:36.276587 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.276552 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:36.350234 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.350197 4828 generic.go:356] "Generic (PLEG): container finished" podID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerID="8890b723f62f356a50ac52dc584942286e61483bca974a2b5c0268088a9ce2a9" exitCode=0 Oct 30 17:06:36.350417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.350259 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" Oct 30 17:06:36.350417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.350281 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" event={"ID":"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0","Type":"ContainerDied","Data":"8890b723f62f356a50ac52dc584942286e61483bca974a2b5c0268088a9ce2a9"} Oct 30 17:06:36.350417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.350349 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn" event={"ID":"a2fa6a03-4ccd-4e0a-81de-9e406e1894a0","Type":"ContainerDied","Data":"c8c526b9b6239f3611e1902baa7e807375d66693c3126702a9cf9c7784c80e56"} Oct 30 17:06:36.350417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.350375 4828 scope.go:122] "RemoveContainer" containerID="8890b723f62f356a50ac52dc584942286e61483bca974a2b5c0268088a9ce2a9" Oct 30 17:06:36.362019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.361999 4828 scope.go:122] "RemoveContainer" containerID="b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508" Oct 30 17:06:36.373078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.373056 4828 scope.go:122] "RemoveContainer" containerID="8890b723f62f356a50ac52dc584942286e61483bca974a2b5c0268088a9ce2a9" Oct 30 17:06:36.373380 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:06:36.373361 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8890b723f62f356a50ac52dc584942286e61483bca974a2b5c0268088a9ce2a9\": container with ID starting with 8890b723f62f356a50ac52dc584942286e61483bca974a2b5c0268088a9ce2a9 not found: ID does not exist" containerID="8890b723f62f356a50ac52dc584942286e61483bca974a2b5c0268088a9ce2a9" Oct 30 17:06:36.373472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.373387 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8890b723f62f356a50ac52dc584942286e61483bca974a2b5c0268088a9ce2a9"} err="failed to get container status \"8890b723f62f356a50ac52dc584942286e61483bca974a2b5c0268088a9ce2a9\": rpc error: code = NotFound desc = could not find container \"8890b723f62f356a50ac52dc584942286e61483bca974a2b5c0268088a9ce2a9\": container with ID starting with 8890b723f62f356a50ac52dc584942286e61483bca974a2b5c0268088a9ce2a9 not found: ID does not exist" Oct 30 17:06:36.373472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.373405 4828 scope.go:122] "RemoveContainer" containerID="b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508" Oct 30 17:06:36.373630 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:06:36.373609 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508\": container with ID starting with b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508 not found: ID does not exist" containerID="b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508" Oct 30 17:06:36.373696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.373635 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508"} err="failed to get container status \"b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508\": rpc error: code = NotFound desc = could not find container \"b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508\": container with ID starting with b8a2f05ea7fa2de54e487374914f7c7b8e85695ff796def9a1df0d598a954508 not found: ID does not exist" Oct 30 17:06:36.423629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.423584 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn"] Oct 30 17:06:36.435005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.434962 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7eda4908-openshift-default-5dc7cd6bb9-hw6pn"] Oct 30 17:06:36.496859 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.496820 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9467d3b9-f7da-43e8-a841-e20944fbb3fe" path="/var/lib/kubelet/pods/9467d3b9-f7da-43e8-a841-e20944fbb3fe/volumes" Oct 30 17:06:36.497638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.497615 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" path="/var/lib/kubelet/pods/a2fa6a03-4ccd-4e0a-81de-9e406e1894a0/volumes" Oct 30 17:06:36.713387 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.713272 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:36.853668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.853626 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852"] Oct 30 17:06:36.854421 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.854405 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" Oct 30 17:06:36.854494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.854422 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" Oct 30 17:06:36.854494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.854450 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" Oct 30 17:06:36.854494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.854455 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" Oct 30 17:06:36.854494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.854469 4828 cpu_manager.go:414] "containerMap: removing container" podUID="9467d3b9-f7da-43e8-a841-e20944fbb3fe" containerName="maas-api" Oct 30 17:06:36.854494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.854474 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9467d3b9-f7da-43e8-a841-e20944fbb3fe" containerName="maas-api" Oct 30 17:06:36.854643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.854585 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" Oct 30 17:06:36.854643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.854596 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fa6a03-4ccd-4e0a-81de-9e406e1894a0" containerName="istio-proxy" Oct 30 17:06:36.854643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.854604 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="9467d3b9-f7da-43e8-a841-e20944fbb3fe" containerName="maas-api" Oct 30 17:06:36.859700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.859677 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" Oct 30 17:06:36.869532 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.869501 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:36.886720 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.879674 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4aba8b6b-dockercfg-7tp6m\"" Oct 30 17:06:36.886990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.883732 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4aba8b6b\"" Oct 30 17:06:36.887093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:36.885032 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852"] Oct 30 17:06:37.119936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:37.093070 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1096e5-26fe-4714-aa64-1998907aca7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852\" (UID: \"8c1096e5-26fe-4714-aa64-1998907aca7b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" Oct 30 17:06:37.119936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:37.093295 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4lxx\" (UniqueName: \"kubernetes.io/projected/8c1096e5-26fe-4714-aa64-1998907aca7b-kube-api-access-f4lxx\") pod \"payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852\" (UID: \"8c1096e5-26fe-4714-aa64-1998907aca7b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" Oct 30 17:06:37.202503 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:37.202460 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4lxx\" (UniqueName: \"kubernetes.io/projected/8c1096e5-26fe-4714-aa64-1998907aca7b-kube-api-access-f4lxx\") pod \"payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852\" (UID: \"8c1096e5-26fe-4714-aa64-1998907aca7b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" Oct 30 17:06:37.202757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:37.202732 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1096e5-26fe-4714-aa64-1998907aca7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852\" (UID: \"8c1096e5-26fe-4714-aa64-1998907aca7b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" Oct 30 17:06:37.204893 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:37.203854 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1096e5-26fe-4714-aa64-1998907aca7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852\" (UID: \"8c1096e5-26fe-4714-aa64-1998907aca7b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" Oct 30 17:06:37.426700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:37.426622 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4lxx\" (UniqueName: \"kubernetes.io/projected/8c1096e5-26fe-4714-aa64-1998907aca7b-kube-api-access-f4lxx\") pod \"payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852\" (UID: \"8c1096e5-26fe-4714-aa64-1998907aca7b\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" Oct 30 17:06:37.491175 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:37.491144 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" Oct 30 17:06:37.816969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:37.816570 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:37.958622 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:06:37.956451 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1096e5_26fe_4714_aa64_1998907aca7b.slice/crio-217ffe2219a9b816ce96ff846ca99ee55aa0eb8a96e11accb879fb4aa6f167c5 WatchSource:0}: Error finding container 217ffe2219a9b816ce96ff846ca99ee55aa0eb8a96e11accb879fb4aa6f167c5: Status 404 returned error can't find the container with id 217ffe2219a9b816ce96ff846ca99ee55aa0eb8a96e11accb879fb4aa6f167c5 Oct 30 17:06:37.958622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:37.956504 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852"] Oct 30 17:06:37.988169 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:06:37.988101 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:06:38.502784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:38.502748 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" event={"ID":"8c1096e5-26fe-4714-aa64-1998907aca7b","Type":"ContainerStarted","Data":"90a8efa3d1e882a790a56df42a55a0a30fb3e04c3aa07b3a908f3a4134070c29"} Oct 30 17:06:38.503021 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:38.502808 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" event={"ID":"8c1096e5-26fe-4714-aa64-1998907aca7b","Type":"ContainerStarted","Data":"217ffe2219a9b816ce96ff846ca99ee55aa0eb8a96e11accb879fb4aa6f167c5"} Oct 30 17:06:38.503093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:38.503028 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" Oct 30 17:06:38.546619 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:38.546567 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" podStartSLOduration=2.546553163 podStartE2EDuration="2.546553163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:38.544220739 +0000 UTC m=+1176.189730909" watchObservedRunningTime="2026-10-30 17:06:38.546553163 +0000 UTC m=+1176.192063342" Oct 30 17:06:39.126595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:39.126558 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:39.300542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:39.300451 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:39.476349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:39.476257 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:39.799061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:39.787635 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:40.242398 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.242374 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:06:40.340192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.340089 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04832433-d5db-4505-83ca-24001a6b2350-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04832433-d5db-4505-83ca-24001a6b2350-istiod-ca-cert\") pod \"04832433-d5db-4505-83ca-24001a6b2350\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " Oct 30 17:06:40.340374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.340216 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-data\") pod \"04832433-d5db-4505-83ca-24001a6b2350\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " Oct 30 17:06:40.340374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.340272 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-envoy\") pod \"04832433-d5db-4505-83ca-24001a6b2350\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " Oct 30 17:06:40.340498 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.340455 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-socket\") pod \"04832433-d5db-4505-83ca-24001a6b2350\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " Oct 30 17:06:40.340556 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.340493 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-credential-socket\") pod \"04832433-d5db-4505-83ca-24001a6b2350\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " Oct 30 17:06:40.340556 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.340531 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-istio-token\" (UniqueName: \"kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-istio-token\") pod \"04832433-d5db-4505-83ca-24001a6b2350\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " Oct 30 17:06:40.340556 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.340532 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04832433-d5db-4505-83ca-24001a6b2350-istiod-ca-cert" pod "04832433-d5db-4505-83ca-24001a6b2350" (UID: "04832433-d5db-4505-83ca-24001a6b2350"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:40.340708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.340596 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-kube-api-access-fqtq6\" (UniqueName: \"kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-kube-api-access-fqtq6\") pod \"04832433-d5db-4505-83ca-24001a6b2350\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " Oct 30 17:06:40.340708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.340650 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-certs\") pod \"04832433-d5db-4505-83ca-24001a6b2350\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " Oct 30 17:06:40.340708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.340688 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04832433-d5db-4505-83ca-24001a6b2350-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04832433-d5db-4505-83ca-24001a6b2350-istio-podinfo\") pod \"04832433-d5db-4505-83ca-24001a6b2350\" (UID: \"04832433-d5db-4505-83ca-24001a6b2350\") " Oct 30 17:06:40.341150 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.341122 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-credential-socket" pod "04832433-d5db-4505-83ca-24001a6b2350" (UID: "04832433-d5db-4505-83ca-24001a6b2350"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:40.341323 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.341233 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-certs" pod "04832433-d5db-4505-83ca-24001a6b2350" (UID: "04832433-d5db-4505-83ca-24001a6b2350"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:40.341400 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.341366 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-socket" pod "04832433-d5db-4505-83ca-24001a6b2350" (UID: "04832433-d5db-4505-83ca-24001a6b2350"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:40.341456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.341424 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-data" pod "04832433-d5db-4505-83ca-24001a6b2350" (UID: "04832433-d5db-4505-83ca-24001a6b2350"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:40.341712 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.341690 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.341845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.341831 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.342019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.342003 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.342114 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.342102 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.342205 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.342193 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04832433-d5db-4505-83ca-24001a6b2350-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.344055 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.344029 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04832433-d5db-4505-83ca-24001a6b2350-istio-podinfo" pod "04832433-d5db-4505-83ca-24001a6b2350" (UID: "04832433-d5db-4505-83ca-24001a6b2350"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:06:40.344163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.344089 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-envoy" pod "04832433-d5db-4505-83ca-24001a6b2350" (UID: "04832433-d5db-4505-83ca-24001a6b2350"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:40.346319 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.346282 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-istio-token" pod "04832433-d5db-4505-83ca-24001a6b2350" (UID: "04832433-d5db-4505-83ca-24001a6b2350"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:40.347587 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.347552 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-kube-api-access-fqtq6" pod "04832433-d5db-4505-83ca-24001a6b2350" (UID: "04832433-d5db-4505-83ca-24001a6b2350"). InnerVolumeSpecName "kube-api-access-fqtq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:40.443871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.443837 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04832433-d5db-4505-83ca-24001a6b2350-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.443871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.443866 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.443871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.443875 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqtq6\" (UniqueName: \"kubernetes.io/projected/04832433-d5db-4505-83ca-24001a6b2350-kube-api-access-fqtq6\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.444120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.443885 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04832433-d5db-4505-83ca-24001a6b2350-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:06:40.524293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.524259 4828 generic.go:356] "Generic (PLEG): container finished" podID="04832433-d5db-4505-83ca-24001a6b2350" containerID="c993f833c41415a74e826ae39d886b670cd693827180f0c3cee2d772280153ad" exitCode=0 Oct 30 17:06:40.524470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.524326 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" Oct 30 17:06:40.524470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.524345 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" event={"ID":"04832433-d5db-4505-83ca-24001a6b2350","Type":"ContainerDied","Data":"c993f833c41415a74e826ae39d886b670cd693827180f0c3cee2d772280153ad"} Oct 30 17:06:40.524470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.524401 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq" event={"ID":"04832433-d5db-4505-83ca-24001a6b2350","Type":"ContainerDied","Data":"e480f1fb6e3549634b8d68b3b60008fb5ad89fdc396d0abad06f56b694e97565"} Oct 30 17:06:40.524470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.524429 4828 scope.go:122] "RemoveContainer" containerID="c993f833c41415a74e826ae39d886b670cd693827180f0c3cee2d772280153ad" Oct 30 17:06:40.535974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.535955 4828 scope.go:122] "RemoveContainer" containerID="cf5a4c2e9b7c0e7410a12fe3bcd1d9983bab44ffc4f29a73491cb6797e6c882e" Oct 30 17:06:40.563928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.548353 4828 scope.go:122] "RemoveContainer" containerID="c993f833c41415a74e826ae39d886b670cd693827180f0c3cee2d772280153ad" Oct 30 17:06:40.563928 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:06:40.549303 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c993f833c41415a74e826ae39d886b670cd693827180f0c3cee2d772280153ad\": container with ID starting with c993f833c41415a74e826ae39d886b670cd693827180f0c3cee2d772280153ad not found: ID does not exist" containerID="c993f833c41415a74e826ae39d886b670cd693827180f0c3cee2d772280153ad" Oct 30 17:06:40.563928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.549347 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c993f833c41415a74e826ae39d886b670cd693827180f0c3cee2d772280153ad"} err="failed to get container status \"c993f833c41415a74e826ae39d886b670cd693827180f0c3cee2d772280153ad\": rpc error: code = NotFound desc = could not find container \"c993f833c41415a74e826ae39d886b670cd693827180f0c3cee2d772280153ad\": container with ID starting with c993f833c41415a74e826ae39d886b670cd693827180f0c3cee2d772280153ad not found: ID does not exist" Oct 30 17:06:40.563928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.549376 4828 scope.go:122] "RemoveContainer" containerID="cf5a4c2e9b7c0e7410a12fe3bcd1d9983bab44ffc4f29a73491cb6797e6c882e" Oct 30 17:06:40.563928 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:06:40.549701 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5a4c2e9b7c0e7410a12fe3bcd1d9983bab44ffc4f29a73491cb6797e6c882e\": container with ID starting with cf5a4c2e9b7c0e7410a12fe3bcd1d9983bab44ffc4f29a73491cb6797e6c882e not found: ID does not exist" containerID="cf5a4c2e9b7c0e7410a12fe3bcd1d9983bab44ffc4f29a73491cb6797e6c882e" Oct 30 17:06:40.563928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.549728 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5a4c2e9b7c0e7410a12fe3bcd1d9983bab44ffc4f29a73491cb6797e6c882e"} err="failed to get container status \"cf5a4c2e9b7c0e7410a12fe3bcd1d9983bab44ffc4f29a73491cb6797e6c882e\": rpc error: code = NotFound desc = could not find container \"cf5a4c2e9b7c0e7410a12fe3bcd1d9983bab44ffc4f29a73491cb6797e6c882e\": container with ID starting with cf5a4c2e9b7c0e7410a12fe3bcd1d9983bab44ffc4f29a73491cb6797e6c882e not found: ID does not exist" Oct 30 17:06:40.726987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.726332 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq"] Oct 30 17:06:40.833924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:40.828517 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-952bc16e-openshift-default-98bff8d4c-sqstq"] Oct 30 17:06:42.273941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:42.263015 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:42.489923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:42.489628 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:42.618395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:42.618313 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04832433-d5db-4505-83ca-24001a6b2350" path="/var/lib/kubelet/pods/04832433-d5db-4505-83ca-24001a6b2350/volumes" Oct 30 17:06:42.674920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:42.671475 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:44.346921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:44.340650 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:45.235736 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:45.180602 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:45.369069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:45.369016 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:45.455948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:45.396889 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:06:45.720945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:45.711300 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:06:45.720945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:45.711387 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" podUID="748eedeb-7a6c-4bc9-b141-29be1431024a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:45.755953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:45.722627 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:46.045017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:46.014570 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:46.130923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:46.112198 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:46.166559 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:46.166512 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:46.346083 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:46.345985 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:06:46.386926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:46.352648 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": context deadline exceeded" Oct 30 17:06:46.876222 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:46.876187 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:06:47.077943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:46.943983 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" podUID="08423875-76fe-41f6-995a-6c5832230d12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:06:47.136876 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:47.136374 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:47.465262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:47.457926 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:47.701341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:47.701287 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:48.374414 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:48.374010 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:49.305922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:49.305181 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:49.517128 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:49.517099 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" Oct 30 17:06:51.173967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:51.165284 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:52.047194 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.047140 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:52.437502 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.437399 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq"] Oct 30 17:06:52.438695 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.438670 4828 cpu_manager.go:414] "containerMap: removing container" podUID="04832433-d5db-4505-83ca-24001a6b2350" containerName="istio-proxy" Oct 30 17:06:52.438789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.438698 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04832433-d5db-4505-83ca-24001a6b2350" containerName="istio-proxy" Oct 30 17:06:52.438789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.438736 4828 cpu_manager.go:414] "containerMap: removing container" podUID="04832433-d5db-4505-83ca-24001a6b2350" containerName="istio-proxy" Oct 30 17:06:52.438789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.438744 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04832433-d5db-4505-83ca-24001a6b2350" containerName="istio-proxy" Oct 30 17:06:52.439001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.438931 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="04832433-d5db-4505-83ca-24001a6b2350" containerName="istio-proxy" Oct 30 17:06:52.439001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.438950 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="04832433-d5db-4505-83ca-24001a6b2350" containerName="istio-proxy" Oct 30 17:06:52.443508 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.443485 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.461659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.461610 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq"] Oct 30 17:06:52.468629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.468603 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-11f480-openshift-default-dockercfg-q54n7\"" Oct 30 17:06:52.512480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.512449 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:52.558640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.558608 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.558814 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.558656 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383c4614-f032-4952-9be1-d69075b86db8-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.558814 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.558688 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.558939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.558814 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.558939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.558850 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.558939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.558876 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jcp\" (UniqueName: \"kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-kube-api-access-f5jcp\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.559075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.558979 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.559075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.559009 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383c4614-f032-4952-9be1-d69075b86db8-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.559201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.559178 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.674093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.662996 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.674093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.663075 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383c4614-f032-4952-9be1-d69075b86db8-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.674093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.663125 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.674093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.663257 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.674093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.663309 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.674093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.663343 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jcp\" (UniqueName: \"kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-kube-api-access-f5jcp\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.674093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.663382 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.674093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.663405 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383c4614-f032-4952-9be1-d69075b86db8-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.674093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.663542 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.674093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.664628 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.704963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.699663 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383c4614-f032-4952-9be1-d69075b86db8-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.704963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.699996 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.719920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.718687 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.719920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.719364 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383c4614-f032-4952-9be1-d69075b86db8-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.719920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.719602 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.720222 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.720039 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.767923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.753623 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jcp\" (UniqueName: \"kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-kube-api-access-f5jcp\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.767923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.760496 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.767923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.761076 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:52.784275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.784236 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:52.978990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:52.977513 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:53.250406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.250308 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:53.414065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.414027 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:53.463377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.463343 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4"] Oct 30 17:06:53.469133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.469109 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:53.551776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.540206 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4aba8b6b-serving-cert\"" Oct 30 17:06:53.588939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.578133 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-maas-api-tls\") pod \"maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:53.588939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.578226 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-metrics-certs\") pod \"maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:53.588939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.578297 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f1c516-77d4-4e05-b53d-64fc75b99a5f-openshift-service-ca\") pod \"maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:53.588939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.578424 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrx2m\" (UniqueName: \"kubernetes.io/projected/85f1c516-77d4-4e05-b53d-64fc75b99a5f-kube-api-access-jrx2m\") pod \"maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:53.619561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.619417 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:53.629958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.626525 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4"] Oct 30 17:06:53.684369 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.684330 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-maas-api-tls\") pod \"maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:53.684915 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.684877 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-metrics-certs\") pod \"maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:53.694109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.694079 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f1c516-77d4-4e05-b53d-64fc75b99a5f-openshift-service-ca\") pod \"maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:53.694443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.694418 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrx2m\" (UniqueName: \"kubernetes.io/projected/85f1c516-77d4-4e05-b53d-64fc75b99a5f-kube-api-access-jrx2m\") pod \"maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:53.695128 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.693838 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq"] Oct 30 17:06:53.696168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.695108 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-maas-api-tls\") pod \"maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:53.696296 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.695706 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f1c516-77d4-4e05-b53d-64fc75b99a5f-openshift-service-ca\") pod \"maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:53.738920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.721516 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-metrics-certs\") pod \"maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:53.743802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.743766 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrx2m\" (UniqueName: \"kubernetes.io/projected/85f1c516-77d4-4e05-b53d-64fc75b99a5f-kube-api-access-jrx2m\") pod \"maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:53.748710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.748625 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:53.748952 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.748931 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:53.749088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.749069 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:06:53.839981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.831805 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:53.942935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:53.942816 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:54.016188 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:54.015053 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:54.420146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:54.420046 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" event={"ID":"383c4614-f032-4952-9be1-d69075b86db8","Type":"ContainerStarted","Data":"81e498bc4f22ef97f4e982dec3ad06d920c8ac1f8f8728ab29e2e5cf6a701c8c"} Oct 30 17:06:54.420146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:54.420117 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" event={"ID":"383c4614-f032-4952-9be1-d69075b86db8","Type":"ContainerStarted","Data":"e0d369f3c4d3fd96ae0ecb4e9ab6ce96d696b91fc18f5ae012a57def7dded41d"} Oct 30 17:06:54.491164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:54.491112 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" podStartSLOduration=2.491094937 podStartE2EDuration="2.491094937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:54.485411701 +0000 UTC m=+1192.130921882" watchObservedRunningTime="2026-10-30 17:06:54.491094937 +0000 UTC m=+1192.136605117" Oct 30 17:06:54.530289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:54.530248 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:54.547284 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:06:54.547250 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f1c516_77d4_4e05_b53d_64fc75b99a5f.slice/crio-2a6a97a43e2c6a7ae21db1ceb15734c63cd92130e769d5d1bfba4ba0b82980ce WatchSource:0}: Error finding container 2a6a97a43e2c6a7ae21db1ceb15734c63cd92130e769d5d1bfba4ba0b82980ce: Status 404 returned error can't find the container with id 2a6a97a43e2c6a7ae21db1ceb15734c63cd92130e769d5d1bfba4ba0b82980ce Oct 30 17:06:54.550406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:54.550380 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4"] Oct 30 17:06:54.785927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:54.785469 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:54.789935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:54.786278 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:55.031396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:55.029027 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:55.254928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:55.254112 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:55.578232 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:55.578187 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" event={"ID":"85f1c516-77d4-4e05-b53d-64fc75b99a5f","Type":"ContainerStarted","Data":"2a6a97a43e2c6a7ae21db1ceb15734c63cd92130e769d5d1bfba4ba0b82980ce"} Oct 30 17:06:55.578696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:55.578295 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:06:55.623625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:55.623572 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" podStartSLOduration=2.050559876 podStartE2EDuration="2.623559499s" totalImagesPullingTime="572.999623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:55.622871294 +0000 UTC m=+1193.268381475" watchObservedRunningTime="2026-10-30 17:06:55.623559499 +0000 UTC m=+1193.269069679" Oct 30 17:06:55.797223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:55.797185 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:55.871757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:55.871709 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:56.074047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:56.071869 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:56.356337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:56.356262 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:56.582317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:56.582280 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" event={"ID":"85f1c516-77d4-4e05-b53d-64fc75b99a5f","Type":"ContainerStarted","Data":"e2f78e14dac51299e481469f0c1aa3fb60e73f76fe1ea283cc944d0cf9ee7954"} Oct 30 17:06:56.582800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:56.582567 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:56.583780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:56.583758 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:06:57.617531 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:57.617491 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:58.730384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:58.730333 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:06:58.898308 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:06:58.898266 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:00.546807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:00.546769 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:02.395925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:02.392372 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:02.596182 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:02.596152 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:07:03.432383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:03.432345 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:03.914049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:03.914012 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:04.074916 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:04.074794 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:04.374030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:04.373994 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:04.796910 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:04.796859 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn"] Oct 30 17:07:04.797588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:04.797557 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" containerID="cri-o://ec82117210e1cd6b23b620078bfb265ed343114aaf349d8d40d07a6e4876d810" gracePeriod=30 Oct 30 17:07:04.811082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:04.811041 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl"] Oct 30 17:07:04.811372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:04.811338 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" containerID="cri-o://c2dadf7940e2377c11d59e266cd28419726a7bf28c83b55801479ed9430ff466" gracePeriod=30 Oct 30 17:07:04.816303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:04.816277 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l"] Oct 30 17:07:04.816480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:04.816455 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" podUID="0ab34194-be4a-48ba-9b6d-c759be67f22d" containerName="istio-proxy" containerID="cri-o://c158340e8474f363e366ee96ef8532afa83a49e0f1a10dad610f51a9748c1167" gracePeriod=30 Oct 30 17:07:04.979599 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:04.979569 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:07:04.979771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:04.979660 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:07:05.835924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:05.831060 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:06.308622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:06.308577 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-kkhmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:07:06.308827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:06.308664 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:07:06.792269 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:06.792234 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-dccbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:07:06.792472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:06.792309 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:07:07.682619 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:07.680593 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:07.914541 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:07:07.914500 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:07:07.993167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:07.993077 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:08.147761 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:08.147688 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:08.566967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:08.566882 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:09.638756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:09.638712 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:09.788333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:09.788295 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:09.823731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:09.823696 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr"] Oct 30 17:07:09.829880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:09.829826 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" Oct 30 17:07:09.846348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:09.846296 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr"] Oct 30 17:07:09.852039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:09.852012 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-5d8947\"" Oct 30 17:07:09.852227 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:09.852019 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-5d8947-dockercfg-858dh\"" Oct 30 17:07:09.863875 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:09.863843 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7d0036-8373-47e1-98f8-70c094b0e8d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr\" (UID: \"ea7d0036-8373-47e1-98f8-70c094b0e8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" Oct 30 17:07:09.864353 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:09.864326 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf4rx\" (UniqueName: \"kubernetes.io/projected/ea7d0036-8373-47e1-98f8-70c094b0e8d9-kube-api-access-xf4rx\") pod \"payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr\" (UID: \"ea7d0036-8373-47e1-98f8-70c094b0e8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" Oct 30 17:07:09.981984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:09.981597 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7d0036-8373-47e1-98f8-70c094b0e8d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr\" (UID: \"ea7d0036-8373-47e1-98f8-70c094b0e8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" Oct 30 17:07:09.985948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:09.983179 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7d0036-8373-47e1-98f8-70c094b0e8d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr\" (UID: \"ea7d0036-8373-47e1-98f8-70c094b0e8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" Oct 30 17:07:09.985948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:09.981888 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf4rx\" (UniqueName: \"kubernetes.io/projected/ea7d0036-8373-47e1-98f8-70c094b0e8d9-kube-api-access-xf4rx\") pod \"payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr\" (UID: \"ea7d0036-8373-47e1-98f8-70c094b0e8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" Oct 30 17:07:10.054196 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.048747 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8"] Oct 30 17:07:10.145924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.145327 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" Oct 30 17:07:10.198049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.194574 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf4rx\" (UniqueName: \"kubernetes.io/projected/ea7d0036-8373-47e1-98f8-70c094b0e8d9-kube-api-access-xf4rx\") pod \"payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr\" (UID: \"ea7d0036-8373-47e1-98f8-70c094b0e8d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" Oct 30 17:07:10.295247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.279324 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb59754-4f88-4815-bc64-b53876d50b55-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8\" (UID: \"4eb59754-4f88-4815-bc64-b53876d50b55\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" Oct 30 17:07:10.295247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.279664 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whcmf\" (UniqueName: \"kubernetes.io/projected/4eb59754-4f88-4815-bc64-b53876d50b55-kube-api-access-whcmf\") pod \"payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8\" (UID: \"4eb59754-4f88-4815-bc64-b53876d50b55\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" Oct 30 17:07:10.295247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.288602 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8"] Oct 30 17:07:10.424427 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.423358 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb59754-4f88-4815-bc64-b53876d50b55-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8\" (UID: \"4eb59754-4f88-4815-bc64-b53876d50b55\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" Oct 30 17:07:10.424427 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.423617 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whcmf\" (UniqueName: \"kubernetes.io/projected/4eb59754-4f88-4815-bc64-b53876d50b55-kube-api-access-whcmf\") pod \"payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8\" (UID: \"4eb59754-4f88-4815-bc64-b53876d50b55\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" Oct 30 17:07:10.424427 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.424178 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" Oct 30 17:07:10.435012 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.430459 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb59754-4f88-4815-bc64-b53876d50b55-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8\" (UID: \"4eb59754-4f88-4815-bc64-b53876d50b55\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" Oct 30 17:07:10.461558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.461515 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whcmf\" (UniqueName: \"kubernetes.io/projected/4eb59754-4f88-4815-bc64-b53876d50b55-kube-api-access-whcmf\") pod \"payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8\" (UID: \"4eb59754-4f88-4815-bc64-b53876d50b55\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" Oct 30 17:07:10.565568 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.565531 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" Oct 30 17:07:10.623637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.623276 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.633859 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-data\") pod \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.633946 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-podinfo\") pod \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.634101 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-socket\") pod \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.634144 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-token\" (UniqueName: \"kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-token\") pod \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.634184 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-data" pod "c7750e48-b71a-4a12-94b3-f5ab86d3d40c" (UID: "c7750e48-b71a-4a12-94b3-f5ab86d3d40c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.634189 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istiod-ca-cert\") pod \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.634353 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-socket" pod "c7750e48-b71a-4a12-94b3-f5ab86d3d40c" (UID: "c7750e48-b71a-4a12-94b3-f5ab86d3d40c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.634372 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-certs\") pod \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.634426 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-envoy\") pod \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.634520 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-credential-socket\") pod \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.634555 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-kube-api-access-wc52h\" (UniqueName: \"kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-kube-api-access-wc52h\") pod \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\" (UID: \"c7750e48-b71a-4a12-94b3-f5ab86d3d40c\") " Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.634892 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-certs" pod "c7750e48-b71a-4a12-94b3-f5ab86d3d40c" (UID: "c7750e48-b71a-4a12-94b3-f5ab86d3d40c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.635387 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.635402 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.635411 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.635956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.635765 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-credential-socket" pod "c7750e48-b71a-4a12-94b3-f5ab86d3d40c" (UID: "c7750e48-b71a-4a12-94b3-f5ab86d3d40c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:10.636771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.636431 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istiod-ca-cert" pod "c7750e48-b71a-4a12-94b3-f5ab86d3d40c" (UID: "c7750e48-b71a-4a12-94b3-f5ab86d3d40c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:10.639391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.639339 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-podinfo" pod "c7750e48-b71a-4a12-94b3-f5ab86d3d40c" (UID: "c7750e48-b71a-4a12-94b3-f5ab86d3d40c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:07:10.641376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.641336 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-kube-api-access-wc52h" pod "c7750e48-b71a-4a12-94b3-f5ab86d3d40c" (UID: "c7750e48-b71a-4a12-94b3-f5ab86d3d40c"). InnerVolumeSpecName "kube-api-access-wc52h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:10.646278 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.645307 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-envoy" pod "c7750e48-b71a-4a12-94b3-f5ab86d3d40c" (UID: "c7750e48-b71a-4a12-94b3-f5ab86d3d40c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:10.646278 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.645503 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-token" pod "c7750e48-b71a-4a12-94b3-f5ab86d3d40c" (UID: "c7750e48-b71a-4a12-94b3-f5ab86d3d40c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:10.738078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.737871 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.738078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.737991 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.738078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.738010 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.738078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.738022 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.738078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.738035 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc52h\" (UniqueName: \"kubernetes.io/projected/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-kube-api-access-wc52h\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.738078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.738049 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7750e48-b71a-4a12-94b3-f5ab86d3d40c-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.740386 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.740052 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 17:07:10.842338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.838994 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-credential-socket\") pod \"0ab34194-be4a-48ba-9b6d-c759be67f22d\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " Oct 30 17:07:10.842338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.839063 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-kube-api-access-thdfb\" (UniqueName: \"kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-kube-api-access-thdfb\") pod \"0ab34194-be4a-48ba-9b6d-c759be67f22d\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " Oct 30 17:07:10.842338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.839101 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-socket\") pod \"0ab34194-be4a-48ba-9b6d-c759be67f22d\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " Oct 30 17:07:10.842338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.839235 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-podinfo\") pod \"0ab34194-be4a-48ba-9b6d-c759be67f22d\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " Oct 30 17:07:10.842338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.839336 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-certs\") pod \"0ab34194-be4a-48ba-9b6d-c759be67f22d\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " Oct 30 17:07:10.842338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.839398 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab34194-be4a-48ba-9b6d-c759be67f22d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab34194-be4a-48ba-9b6d-c759be67f22d-istiod-ca-cert\") pod \"0ab34194-be4a-48ba-9b6d-c759be67f22d\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " Oct 30 17:07:10.842338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.839485 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-token\" (UniqueName: \"kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-token\") pod \"0ab34194-be4a-48ba-9b6d-c759be67f22d\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " Oct 30 17:07:10.842338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.839566 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-data\") pod \"0ab34194-be4a-48ba-9b6d-c759be67f22d\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " Oct 30 17:07:10.842338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.839672 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-envoy\") pod \"0ab34194-be4a-48ba-9b6d-c759be67f22d\" (UID: \"0ab34194-be4a-48ba-9b6d-c759be67f22d\") " Oct 30 17:07:10.842338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.842184 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-certs" pod "0ab34194-be4a-48ba-9b6d-c759be67f22d" (UID: "0ab34194-be4a-48ba-9b6d-c759be67f22d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:10.843736 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.843643 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab34194-be4a-48ba-9b6d-c759be67f22d-istiod-ca-cert" pod "0ab34194-be4a-48ba-9b6d-c759be67f22d" (UID: "0ab34194-be4a-48ba-9b6d-c759be67f22d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:10.853145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.845187 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-token" pod "0ab34194-be4a-48ba-9b6d-c759be67f22d" (UID: "0ab34194-be4a-48ba-9b6d-c759be67f22d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:10.853145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.845450 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-credential-socket" pod "0ab34194-be4a-48ba-9b6d-c759be67f22d" (UID: "0ab34194-be4a-48ba-9b6d-c759be67f22d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:10.853145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.845690 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-socket" pod "0ab34194-be4a-48ba-9b6d-c759be67f22d" (UID: "0ab34194-be4a-48ba-9b6d-c759be67f22d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:10.853145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.846069 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-data" pod "0ab34194-be4a-48ba-9b6d-c759be67f22d" (UID: "0ab34194-be4a-48ba-9b6d-c759be67f22d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:10.853145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.847927 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-kube-api-access-thdfb" pod "0ab34194-be4a-48ba-9b6d-c759be67f22d" (UID: "0ab34194-be4a-48ba-9b6d-c759be67f22d"). InnerVolumeSpecName "kube-api-access-thdfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:10.853145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.850030 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-envoy" pod "0ab34194-be4a-48ba-9b6d-c759be67f22d" (UID: "0ab34194-be4a-48ba-9b6d-c759be67f22d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:10.853145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.852309 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-podinfo" pod "0ab34194-be4a-48ba-9b6d-c759be67f22d" (UID: "0ab34194-be4a-48ba-9b6d-c759be67f22d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:07:10.854612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.854591 4828 generic.go:356] "Generic (PLEG): container finished" podID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerID="ec82117210e1cd6b23b620078bfb265ed343114aaf349d8d40d07a6e4876d810" exitCode=0 Oct 30 17:07:10.854733 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.854712 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" Oct 30 17:07:10.854873 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.854852 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" event={"ID":"c7750e48-b71a-4a12-94b3-f5ab86d3d40c","Type":"ContainerDied","Data":"ec82117210e1cd6b23b620078bfb265ed343114aaf349d8d40d07a6e4876d810"} Oct 30 17:07:10.854995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.854983 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn" event={"ID":"c7750e48-b71a-4a12-94b3-f5ab86d3d40c","Type":"ContainerDied","Data":"ed00b9d929b27850104c3e47d6e7aef9c7d98b560acab980b7c7d6604bd8bd39"} Oct 30 17:07:10.855114 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.855103 4828 scope.go:122] "RemoveContainer" containerID="ec82117210e1cd6b23b620078bfb265ed343114aaf349d8d40d07a6e4876d810" Oct 30 17:07:10.859499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.859476 4828 generic.go:356] "Generic (PLEG): container finished" podID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerID="c2dadf7940e2377c11d59e266cd28419726a7bf28c83b55801479ed9430ff466" exitCode=0 Oct 30 17:07:10.859673 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.859655 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" event={"ID":"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda","Type":"ContainerDied","Data":"c2dadf7940e2377c11d59e266cd28419726a7bf28c83b55801479ed9430ff466"} Oct 30 17:07:10.869467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.868935 4828 generic.go:356] "Generic (PLEG): container finished" podID="0ab34194-be4a-48ba-9b6d-c759be67f22d" containerID="c158340e8474f363e366ee96ef8532afa83a49e0f1a10dad610f51a9748c1167" exitCode=0 Oct 30 17:07:10.869467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.869143 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" event={"ID":"0ab34194-be4a-48ba-9b6d-c759be67f22d","Type":"ContainerDied","Data":"c158340e8474f363e366ee96ef8532afa83a49e0f1a10dad610f51a9748c1167"} Oct 30 17:07:10.869467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.869190 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" event={"ID":"0ab34194-be4a-48ba-9b6d-c759be67f22d","Type":"ContainerDied","Data":"efb23865fb8ed8c5658f4db5a0125f8b0df3504cb5ce8f68087ebd4820d107d1"} Oct 30 17:07:10.869467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.869274 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l" Oct 30 17:07:10.906331 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.906231 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:07:10.907293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.906760 4828 scope.go:122] "RemoveContainer" containerID="ec82117210e1cd6b23b620078bfb265ed343114aaf349d8d40d07a6e4876d810" Oct 30 17:07:10.907293 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:07:10.907126 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec82117210e1cd6b23b620078bfb265ed343114aaf349d8d40d07a6e4876d810\": container with ID starting with ec82117210e1cd6b23b620078bfb265ed343114aaf349d8d40d07a6e4876d810 not found: ID does not exist" containerID="ec82117210e1cd6b23b620078bfb265ed343114aaf349d8d40d07a6e4876d810" Oct 30 17:07:10.907293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.907160 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec82117210e1cd6b23b620078bfb265ed343114aaf349d8d40d07a6e4876d810"} err="failed to get container status \"ec82117210e1cd6b23b620078bfb265ed343114aaf349d8d40d07a6e4876d810\": rpc error: code = NotFound desc = could not find container \"ec82117210e1cd6b23b620078bfb265ed343114aaf349d8d40d07a6e4876d810\": container with ID starting with ec82117210e1cd6b23b620078bfb265ed343114aaf349d8d40d07a6e4876d810 not found: ID does not exist" Oct 30 17:07:10.907293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.907185 4828 scope.go:122] "RemoveContainer" containerID="c158340e8474f363e366ee96ef8532afa83a49e0f1a10dad610f51a9748c1167" Oct 30 17:07:10.928684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.922344 4828 scope.go:122] "RemoveContainer" containerID="c158340e8474f363e366ee96ef8532afa83a49e0f1a10dad610f51a9748c1167" Oct 30 17:07:10.929164 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:07:10.929071 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c158340e8474f363e366ee96ef8532afa83a49e0f1a10dad610f51a9748c1167\": container with ID starting with c158340e8474f363e366ee96ef8532afa83a49e0f1a10dad610f51a9748c1167 not found: ID does not exist" containerID="c158340e8474f363e366ee96ef8532afa83a49e0f1a10dad610f51a9748c1167" Oct 30 17:07:10.929164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.929131 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c158340e8474f363e366ee96ef8532afa83a49e0f1a10dad610f51a9748c1167"} err="failed to get container status \"c158340e8474f363e366ee96ef8532afa83a49e0f1a10dad610f51a9748c1167\": rpc error: code = NotFound desc = could not find container \"c158340e8474f363e366ee96ef8532afa83a49e0f1a10dad610f51a9748c1167\": container with ID starting with c158340e8474f363e366ee96ef8532afa83a49e0f1a10dad610f51a9748c1167 not found: ID does not exist" Oct 30 17:07:10.947927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.947840 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-envoy\") pod \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " Oct 30 17:07:10.948986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.948173 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-token\" (UniqueName: \"kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-token\") pod \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " Oct 30 17:07:10.949137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.949063 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-data\") pod \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " Oct 30 17:07:10.949207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.949134 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-kube-api-access-7g9bh\" (UniqueName: \"kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-kube-api-access-7g9bh\") pod \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " Oct 30 17:07:10.949266 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.949244 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-podinfo\") pod \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " Oct 30 17:07:10.949327 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.949277 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istiod-ca-cert\") pod \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " Oct 30 17:07:10.949832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.949654 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istiod-ca-cert" pod "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" (UID: "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:10.949832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.949807 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-data" pod "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" (UID: "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:10.949986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.949862 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-credential-socket" pod "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" (UID: "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:10.950753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.949335 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-credential-socket\") pod \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " Oct 30 17:07:10.950753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.950635 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.950753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.950650 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.950753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.950659 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.950753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.950669 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab34194-be4a-48ba-9b6d-c759be67f22d-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.950753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.950676 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.950753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.950684 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.950753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.950692 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.950753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.950702 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.950753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.950710 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.950753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.950718 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.950753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.950726 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thdfb\" (UniqueName: \"kubernetes.io/projected/0ab34194-be4a-48ba-9b6d-c759be67f22d-kube-api-access-thdfb\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.950753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.950735 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab34194-be4a-48ba-9b6d-c759be67f22d-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:10.952183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.952155 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-kube-api-access-7g9bh" pod "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" (UID: "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda"). InnerVolumeSpecName "kube-api-access-7g9bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:10.953557 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.953510 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-podinfo" pod "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" (UID: "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:07:10.953557 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.953511 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-envoy" pod "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" (UID: "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:10.953876 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.953858 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn"] Oct 30 17:07:10.957296 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.957265 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-token" pod "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" (UID: "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:10.962074 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:10.962050 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-kkhmn"] Oct 30 17:07:11.019287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.019021 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr"] Oct 30 17:07:11.045887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.045853 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l"] Oct 30 17:07:11.047504 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.047426 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-5x69l"] Oct 30 17:07:11.052406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.052373 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-certs\") pod \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " Oct 30 17:07:11.052674 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.052646 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-certs" pod "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" (UID: "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:11.052959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.052936 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-socket\") pod \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\" (UID: \"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda\") " Oct 30 17:07:11.053257 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.053182 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-socket" pod "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" (UID: "469bb33d-3a10-4c4b-9ac0-e1f1262e8fda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:11.053957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.053935 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:11.054040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.053961 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:11.054040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.053979 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:11.054040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.053992 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:11.054040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.054004 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:11.054040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.054017 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g9bh\" (UniqueName: \"kubernetes.io/projected/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda-kube-api-access-7g9bh\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:11.138022 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:07:11.137987 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb59754_4f88_4815_bc64_b53876d50b55.slice/crio-ab7e5f561acef04788d19c8b6d6333bc840cfb230abae5550207319415ebaa88 WatchSource:0}: Error finding container ab7e5f561acef04788d19c8b6d6333bc840cfb230abae5550207319415ebaa88: Status 404 returned error can't find the container with id ab7e5f561acef04788d19c8b6d6333bc840cfb230abae5550207319415ebaa88 Oct 30 17:07:11.139839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.139753 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8"] Oct 30 17:07:11.561417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.561369 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:11.880566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.880471 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" event={"ID":"4eb59754-4f88-4815-bc64-b53876d50b55","Type":"ContainerStarted","Data":"bcfff059c4a372fed3f88fc7c7c96a6ccac030c345f3de029b20f9ad601103bb"} Oct 30 17:07:11.881161 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.881132 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" event={"ID":"4eb59754-4f88-4815-bc64-b53876d50b55","Type":"ContainerStarted","Data":"ab7e5f561acef04788d19c8b6d6333bc840cfb230abae5550207319415ebaa88"} Oct 30 17:07:11.881255 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.881178 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" Oct 30 17:07:11.882375 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.882343 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" event={"ID":"ea7d0036-8373-47e1-98f8-70c094b0e8d9","Type":"ContainerStarted","Data":"6c8ed4dd4548809e57332a5260e4f03b002ec29a84d1626eaf67fe47838985bc"} Oct 30 17:07:11.882479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.882402 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" event={"ID":"ea7d0036-8373-47e1-98f8-70c094b0e8d9","Type":"ContainerStarted","Data":"6f91db1cff8fc027adbb9b841b7cca918814723168e66d9e2e496a0ce717d88c"} Oct 30 17:07:11.882660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.882634 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" Oct 30 17:07:11.885264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.885245 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" Oct 30 17:07:11.885349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.885241 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl" event={"ID":"469bb33d-3a10-4c4b-9ac0-e1f1262e8fda","Type":"ContainerDied","Data":"d361246501d067135af941d0034a22ff5de3a2409e925f49381925b624740dbe"} Oct 30 17:07:11.885405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.885372 4828 scope.go:122] "RemoveContainer" containerID="c2dadf7940e2377c11d59e266cd28419726a7bf28c83b55801479ed9430ff466" Oct 30 17:07:11.926454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.926389 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" podStartSLOduration=2.926371102 podStartE2EDuration="2.926371102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:11.922800431 +0000 UTC m=+1209.568310610" watchObservedRunningTime="2026-10-30 17:07:11.926371102 +0000 UTC m=+1209.571881282" Oct 30 17:07:11.967317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:11.967252 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" podStartSLOduration=2.967233679 podStartE2EDuration="2.967233679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:11.964979499 +0000 UTC m=+1209.610489679" watchObservedRunningTime="2026-10-30 17:07:11.967233679 +0000 UTC m=+1209.612743862" Oct 30 17:07:12.010942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:12.010869 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl"] Oct 30 17:07:12.020990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:12.020957 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-dccbl"] Oct 30 17:07:12.495377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:12.495332 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab34194-be4a-48ba-9b6d-c759be67f22d" path="/var/lib/kubelet/pods/0ab34194-be4a-48ba-9b6d-c759be67f22d/volumes" Oct 30 17:07:12.495997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:12.495979 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" path="/var/lib/kubelet/pods/469bb33d-3a10-4c4b-9ac0-e1f1262e8fda/volumes" Oct 30 17:07:12.496545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:12.496526 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" path="/var/lib/kubelet/pods/c7750e48-b71a-4a12-94b3-f5ab86d3d40c/volumes" Oct 30 17:07:13.369676 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.369640 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f"] Oct 30 17:07:13.371660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.371631 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" Oct 30 17:07:13.371814 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.371800 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" Oct 30 17:07:13.373158 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.373139 4828 cpu_manager.go:414] "containerMap: removing container" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" Oct 30 17:07:13.373272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.373260 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" Oct 30 17:07:13.373389 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.373377 4828 cpu_manager.go:414] "containerMap: removing container" podUID="0ab34194-be4a-48ba-9b6d-c759be67f22d" containerName="istio-proxy" Oct 30 17:07:13.373480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.373468 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab34194-be4a-48ba-9b6d-c759be67f22d" containerName="istio-proxy" Oct 30 17:07:13.373788 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.373772 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="469bb33d-3a10-4c4b-9ac0-e1f1262e8fda" containerName="istio-proxy" Oct 30 17:07:13.373891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.373881 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab34194-be4a-48ba-9b6d-c759be67f22d" containerName="istio-proxy" Oct 30 17:07:13.374026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.374014 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7750e48-b71a-4a12-94b3-f5ab86d3d40c" containerName="istio-proxy" Oct 30 17:07:13.383493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.383466 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:13.401210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.400199 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-11f480-serving-cert\"" Oct 30 17:07:13.401210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.401060 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f"] Oct 30 17:07:13.455826 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.455792 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:13.456043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.455844 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zrjd\" (UniqueName: \"kubernetes.io/projected/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-kube-api-access-9zrjd\") pod \"maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:13.456149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.456091 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:13.456207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.456192 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:13.544066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.544029 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:13.557817 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.557763 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:13.557999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.557925 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:13.557999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.557973 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zrjd\" (UniqueName: \"kubernetes.io/projected/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-kube-api-access-9zrjd\") pod \"maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:13.558236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.558219 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:13.559686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.559662 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:13.566937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.562034 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:13.566937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.562184 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:13.593696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.593653 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zrjd\" (UniqueName: \"kubernetes.io/projected/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-kube-api-access-9zrjd\") pod \"maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:13.731985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.729438 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:13.736785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:13.734616 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:14.230679 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:07:14.230646 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58a6f81_69f4_4bcd_bbf9_0bc3c93ab81e.slice/crio-3e33c95fa7142a4388dbb8e641215db3472b1b6cb816dc9382b26e839e0ea867 WatchSource:0}: Error finding container 3e33c95fa7142a4388dbb8e641215db3472b1b6cb816dc9382b26e839e0ea867: Status 404 returned error can't find the container with id 3e33c95fa7142a4388dbb8e641215db3472b1b6cb816dc9382b26e839e0ea867 Oct 30 17:07:14.253211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:14.253176 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f"] Oct 30 17:07:15.172012 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:15.171971 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" event={"ID":"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e","Type":"ContainerStarted","Data":"3e33c95fa7142a4388dbb8e641215db3472b1b6cb816dc9382b26e839e0ea867"} Oct 30 17:07:15.558834 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:15.558794 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:15.709187 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:15.709152 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:16.195308 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:16.195268 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" event={"ID":"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e","Type":"ContainerStarted","Data":"307e2e44aa3bcf2a8af9e9ee1ade7398af2139e0573b2eaa02f6208e958dac76"} Oct 30 17:07:16.196102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:16.196080 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:16.249023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:16.247844 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" podStartSLOduration=2.800576088 podStartE2EDuration="3.247829646s" totalImagesPullingTime="447.253558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:16.242137546 +0000 UTC m=+1213.887647727" watchObservedRunningTime="2026-10-30 17:07:16.247829646 +0000 UTC m=+1213.893339827" Oct 30 17:07:17.528922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:17.528542 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:17.685930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:17.685817 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:18.327927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:18.322537 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:19.171927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:19.168999 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:19.171927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:19.170287 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:19.803087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:19.801504 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:20.186017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:20.172642 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:20.186951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:20.186878 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:07:20.366107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:20.366077 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:07:20.604547 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:20.604448 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:22.266937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:22.266132 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:07:22.391004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:22.389147 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:22.900775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:22.900744 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" Oct 30 17:07:22.901006 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:22.900825 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" Oct 30 17:07:24.203112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.203072 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:24.497566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.497471 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:24.891340 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.891300 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44"] Oct 30 17:07:24.896756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.896728 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:24.916932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.916801 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7ebeb6e1-openshift-default-dockercfg-57wpn\"" Oct 30 17:07:24.937298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.937259 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44"] Oct 30 17:07:24.955273 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.955237 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istiod-ca-cert\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:24.955505 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.955483 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-credential-socket\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:24.955611 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.955547 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvcb9\" (UniqueName: \"kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-kube-api-access-rvcb9\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:24.955672 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.955608 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-token\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:24.955727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.955671 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-envoy\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:24.955784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.955747 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-podinfo\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:24.955840 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.955820 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-socket\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:24.956035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.956014 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-data\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:24.956176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:24.956157 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-certs\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.060932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.049423 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:25.062929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.062872 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-data\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.075958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.066973 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-certs\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.076532 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.076498 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istiod-ca-cert\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.076742 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.076722 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-credential-socket\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.076919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.076883 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvcb9\" (UniqueName: \"kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-kube-api-access-rvcb9\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.077088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.077071 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-token\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.080919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.077216 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-envoy\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.081229 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.081204 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-podinfo\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.101100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.101060 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-socket\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.101586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.063520 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-data\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.101743 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.085424 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istiod-ca-cert\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.101845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.093726 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-credential-socket\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.101963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.072436 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-certs\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.102380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.102354 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-socket\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.163973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.136593 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-podinfo\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.255923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.254974 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:25.268928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.259000 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvcb9\" (UniqueName: \"kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-kube-api-access-rvcb9\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.269111 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.268937 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-envoy\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.292977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.272680 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852"] Oct 30 17:07:25.319361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.319329 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-token\") pod \"e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.567165 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.567127 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:25.598243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.598201 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:25.762928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:25.761656 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:26.218716 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:26.218610 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:26.341819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:26.341779 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44"] Oct 30 17:07:26.417944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:26.384188 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:26.417944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:26.414148 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:26.417944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:26.414429 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:26.760217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:26.760183 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" event={"ID":"836c4dab-6052-49e4-b60e-98bfa0f48b5b","Type":"ContainerStarted","Data":"e2ca8adfbdccf2e68d612c0af1c36e56d7770ab912d1d8bcc4e57af7ea3b65ad"} Oct 30 17:07:26.760413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:26.760245 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" event={"ID":"836c4dab-6052-49e4-b60e-98bfa0f48b5b","Type":"ContainerStarted","Data":"70d48691a9fb20e27525b0b5fd33ec8cd53aad5852e8f47313a5fc313f04648a"} Oct 30 17:07:27.597566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:27.597530 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:27.606087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:27.606055 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:27.655324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:27.655269 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" podStartSLOduration=3.655254695 podStartE2EDuration="3.655254695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:26.812094474 +0000 UTC m=+1224.457604654" watchObservedRunningTime="2026-10-30 17:07:27.655254695 +0000 UTC m=+1225.300764875" Oct 30 17:07:27.772928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:27.769661 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:27.793217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:27.793182 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:07:28.231786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:28.231745 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:28.388292 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:28.388257 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:28.569245 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:28.569191 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:29.405772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:29.405733 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:29.579216 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:29.579179 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:30.297058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:30.296983 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" podUID="8c1096e5-26fe-4714-aa64-1998907aca7b" containerName="payload-pre-processing" containerID="cri-o://90a8efa3d1e882a790a56df42a55a0a30fb3e04c3aa07b3a908f3a4134070c29" gracePeriod=25 Oct 30 17:07:30.556226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:30.556014 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:30.866057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:30.866023 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" Oct 30 17:07:30.867990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:30.867964 4828 generic.go:356] "Generic (PLEG): container finished" podID="8c1096e5-26fe-4714-aa64-1998907aca7b" containerID="90a8efa3d1e882a790a56df42a55a0a30fb3e04c3aa07b3a908f3a4134070c29" exitCode=0 Oct 30 17:07:30.868132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:30.868041 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" event={"ID":"8c1096e5-26fe-4714-aa64-1998907aca7b","Type":"ContainerDied","Data":"90a8efa3d1e882a790a56df42a55a0a30fb3e04c3aa07b3a908f3a4134070c29"} Oct 30 17:07:30.868132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:30.868106 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" event={"ID":"8c1096e5-26fe-4714-aa64-1998907aca7b","Type":"ContainerDied","Data":"217ffe2219a9b816ce96ff846ca99ee55aa0eb8a96e11accb879fb4aa6f167c5"} Oct 30 17:07:30.868228 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:30.868137 4828 scope.go:122] "RemoveContainer" containerID="90a8efa3d1e882a790a56df42a55a0a30fb3e04c3aa07b3a908f3a4134070c29" Oct 30 17:07:30.909439 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:30.909408 4828 scope.go:122] "RemoveContainer" containerID="90a8efa3d1e882a790a56df42a55a0a30fb3e04c3aa07b3a908f3a4134070c29" Oct 30 17:07:30.910010 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:07:30.909985 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a8efa3d1e882a790a56df42a55a0a30fb3e04c3aa07b3a908f3a4134070c29\": container with ID starting with 90a8efa3d1e882a790a56df42a55a0a30fb3e04c3aa07b3a908f3a4134070c29 not found: ID does not exist" containerID="90a8efa3d1e882a790a56df42a55a0a30fb3e04c3aa07b3a908f3a4134070c29" Oct 30 17:07:30.910207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:30.910018 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a8efa3d1e882a790a56df42a55a0a30fb3e04c3aa07b3a908f3a4134070c29"} err="failed to get container status \"90a8efa3d1e882a790a56df42a55a0a30fb3e04c3aa07b3a908f3a4134070c29\": rpc error: code = NotFound desc = could not find container \"90a8efa3d1e882a790a56df42a55a0a30fb3e04c3aa07b3a908f3a4134070c29\": container with ID starting with 90a8efa3d1e882a790a56df42a55a0a30fb3e04c3aa07b3a908f3a4134070c29 not found: ID does not exist" Oct 30 17:07:30.950544 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:30.950508 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1096e5-26fe-4714-aa64-1998907aca7b-kube-api-access-f4lxx\" (UniqueName: \"kubernetes.io/projected/8c1096e5-26fe-4714-aa64-1998907aca7b-kube-api-access-f4lxx\") pod \"8c1096e5-26fe-4714-aa64-1998907aca7b\" (UID: \"8c1096e5-26fe-4714-aa64-1998907aca7b\") " Oct 30 17:07:30.950741 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:30.950609 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1096e5-26fe-4714-aa64-1998907aca7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1096e5-26fe-4714-aa64-1998907aca7b-plugins-config-volume\") pod \"8c1096e5-26fe-4714-aa64-1998907aca7b\" (UID: \"8c1096e5-26fe-4714-aa64-1998907aca7b\") " Oct 30 17:07:30.953851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:30.953825 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1096e5-26fe-4714-aa64-1998907aca7b-kube-api-access-f4lxx" pod "8c1096e5-26fe-4714-aa64-1998907aca7b" (UID: "8c1096e5-26fe-4714-aa64-1998907aca7b"). InnerVolumeSpecName "kube-api-access-f4lxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:30.956833 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:30.956799 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1096e5-26fe-4714-aa64-1998907aca7b-plugins-config-volume" pod "8c1096e5-26fe-4714-aa64-1998907aca7b" (UID: "8c1096e5-26fe-4714-aa64-1998907aca7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:31.053244 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.053209 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4lxx\" (UniqueName: \"kubernetes.io/projected/8c1096e5-26fe-4714-aa64-1998907aca7b-kube-api-access-f4lxx\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:31.053244 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.053242 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1096e5-26fe-4714-aa64-1998907aca7b-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:31.337350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.337297 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:31.441579 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.441539 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6"] Oct 30 17:07:31.442640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.442616 4828 cpu_manager.go:414] "containerMap: removing container" podUID="8c1096e5-26fe-4714-aa64-1998907aca7b" containerName="payload-pre-processing" Oct 30 17:07:31.442640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.442637 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1096e5-26fe-4714-aa64-1998907aca7b" containerName="payload-pre-processing" Oct 30 17:07:31.442838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.442818 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1096e5-26fe-4714-aa64-1998907aca7b" containerName="payload-pre-processing" Oct 30 17:07:31.448464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.448444 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" Oct 30 17:07:31.459201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.459176 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4740747b-0ca9-4643-910f-32507d5d0bed-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6\" (UID: \"4740747b-0ca9-4643-910f-32507d5d0bed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" Oct 30 17:07:31.459369 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.459341 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6rfb\" (UniqueName: \"kubernetes.io/projected/4740747b-0ca9-4643-910f-32507d5d0bed-kube-api-access-v6rfb\") pod \"payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6\" (UID: \"4740747b-0ca9-4643-910f-32507d5d0bed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" Oct 30 17:07:31.463989 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.463861 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7ebeb6e1\"" Oct 30 17:07:31.466606 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.466581 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6"] Oct 30 17:07:31.492925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.473142 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7ebeb6e1-dockercfg-zhc9h\"" Oct 30 17:07:31.505953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.502366 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:31.568930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.566927 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4740747b-0ca9-4643-910f-32507d5d0bed-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6\" (UID: \"4740747b-0ca9-4643-910f-32507d5d0bed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" Oct 30 17:07:31.568930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.567146 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6rfb\" (UniqueName: \"kubernetes.io/projected/4740747b-0ca9-4643-910f-32507d5d0bed-kube-api-access-v6rfb\") pod \"payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6\" (UID: \"4740747b-0ca9-4643-910f-32507d5d0bed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" Oct 30 17:07:31.637973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.637389 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4740747b-0ca9-4643-910f-32507d5d0bed-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6\" (UID: \"4740747b-0ca9-4643-910f-32507d5d0bed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" Oct 30 17:07:31.744924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.744152 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw"] Oct 30 17:07:31.809608 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:31.799284 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" Oct 30 17:07:32.116988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.033035 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6rfb\" (UniqueName: \"kubernetes.io/projected/4740747b-0ca9-4643-910f-32507d5d0bed-kube-api-access-v6rfb\") pod \"payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6\" (UID: \"4740747b-0ca9-4643-910f-32507d5d0bed\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" Oct 30 17:07:32.116988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.061976 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw"] Oct 30 17:07:32.116988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.066580 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" Oct 30 17:07:32.116988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.067959 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw\" (UID: \"cdb4b388-a68c-454b-92fc-34a9c1b94a9f\") " pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" Oct 30 17:07:32.116988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.068211 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5jtd\" (UniqueName: \"kubernetes.io/projected/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-kube-api-access-b5jtd\") pod \"payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw\" (UID: \"cdb4b388-a68c-454b-92fc-34a9c1b94a9f\") " pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" Oct 30 17:07:32.116988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.089675 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852" Oct 30 17:07:32.180954 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.179683 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5jtd\" (UniqueName: \"kubernetes.io/projected/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-kube-api-access-b5jtd\") pod \"payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw\" (UID: \"cdb4b388-a68c-454b-92fc-34a9c1b94a9f\") " pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" Oct 30 17:07:32.180954 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.179939 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw\" (UID: \"cdb4b388-a68c-454b-92fc-34a9c1b94a9f\") " pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" Oct 30 17:07:32.182770 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.182716 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw\" (UID: \"cdb4b388-a68c-454b-92fc-34a9c1b94a9f\") " pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" Oct 30 17:07:32.214348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.214320 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5jtd\" (UniqueName: \"kubernetes.io/projected/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-kube-api-access-b5jtd\") pod \"payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw\" (UID: \"cdb4b388-a68c-454b-92fc-34a9c1b94a9f\") " pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" Oct 30 17:07:32.236348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.236175 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852"] Oct 30 17:07:32.244118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.244083 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-4aba8b6b-65795d8d9f-sq852"] Oct 30 17:07:32.442074 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.441978 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" Oct 30 17:07:32.531232 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.531091 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:32.531232 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.531215 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6"] Oct 30 17:07:32.657244 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:07:32.654403 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4740747b_0ca9_4643_910f_32507d5d0bed.slice/crio-0db9403765edcb5b595380c4f8b5c2c1598a4932349d9b2e3472d552ce8439ff WatchSource:0}: Error finding container 0db9403765edcb5b595380c4f8b5c2c1598a4932349d9b2e3472d552ce8439ff: Status 404 returned error can't find the container with id 0db9403765edcb5b595380c4f8b5c2c1598a4932349d9b2e3472d552ce8439ff Oct 30 17:07:32.657244 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.656354 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1096e5-26fe-4714-aa64-1998907aca7b" path="/var/lib/kubelet/pods/8c1096e5-26fe-4714-aa64-1998907aca7b/volumes" Oct 30 17:07:32.992617 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:07:32.992577 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb4b388_a68c_454b_92fc_34a9c1b94a9f.slice/crio-78c96f821b090159484a77558dd1911fb11eee24599385e3e70f5fd9d18a674d WatchSource:0}: Error finding container 78c96f821b090159484a77558dd1911fb11eee24599385e3e70f5fd9d18a674d: Status 404 returned error can't find the container with id 78c96f821b090159484a77558dd1911fb11eee24599385e3e70f5fd9d18a674d Oct 30 17:07:32.995287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:32.995263 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw"] Oct 30 17:07:33.100285 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:33.100247 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" event={"ID":"cdb4b388-a68c-454b-92fc-34a9c1b94a9f","Type":"ContainerStarted","Data":"94e641bcbde48266b4a025e26f0e0b786caa742d962235adef8390b2c1b2c2a6"} Oct 30 17:07:33.100476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:33.100317 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" Oct 30 17:07:33.100476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:33.100342 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" event={"ID":"cdb4b388-a68c-454b-92fc-34a9c1b94a9f","Type":"ContainerStarted","Data":"78c96f821b090159484a77558dd1911fb11eee24599385e3e70f5fd9d18a674d"} Oct 30 17:07:33.102408 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:33.102345 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" event={"ID":"4740747b-0ca9-4643-910f-32507d5d0bed","Type":"ContainerStarted","Data":"3c26742939ff7b68081c51a7f4e22c4705382f4281cff7152e1df85645298fac"} Oct 30 17:07:33.102408 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:33.102404 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" event={"ID":"4740747b-0ca9-4643-910f-32507d5d0bed","Type":"ContainerStarted","Data":"0db9403765edcb5b595380c4f8b5c2c1598a4932349d9b2e3472d552ce8439ff"} Oct 30 17:07:33.102607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:33.102552 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" Oct 30 17:07:33.149975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:33.149894 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" podStartSLOduration=2.149871324 podStartE2EDuration="2.149871324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:33.144853628 +0000 UTC m=+1230.790363807" watchObservedRunningTime="2026-10-30 17:07:33.149871324 +0000 UTC m=+1230.795381505" Oct 30 17:07:33.194563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:33.194489 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" podStartSLOduration=2.19447012 podStartE2EDuration="2.19447012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:33.187020979 +0000 UTC m=+1230.832531160" watchObservedRunningTime="2026-10-30 17:07:33.19447012 +0000 UTC m=+1230.839980301" Oct 30 17:07:33.926259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:33.926221 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:33.976827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:33.976789 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4"] Oct 30 17:07:33.977045 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:33.977019 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" podUID="85f1c516-77d4-4e05-b53d-64fc75b99a5f" containerName="maas-api" containerID="cri-o://e2f78e14dac51299e481469f0c1aa3fb60e73f76fe1ea283cc944d0cf9ee7954" gracePeriod=30 Oct 30 17:07:34.072499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.072460 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:34.113194 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.113162 4828 generic.go:356] "Generic (PLEG): container finished" podID="85f1c516-77d4-4e05-b53d-64fc75b99a5f" containerID="e2f78e14dac51299e481469f0c1aa3fb60e73f76fe1ea283cc944d0cf9ee7954" exitCode=0 Oct 30 17:07:34.113370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.113217 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" event={"ID":"85f1c516-77d4-4e05-b53d-64fc75b99a5f","Type":"ContainerDied","Data":"e2f78e14dac51299e481469f0c1aa3fb60e73f76fe1ea283cc944d0cf9ee7954"} Oct 30 17:07:34.560064 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.559649 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:07:34.738321 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.719491 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f1c516-77d4-4e05-b53d-64fc75b99a5f-kube-api-access-jrx2m\" (UniqueName: \"kubernetes.io/projected/85f1c516-77d4-4e05-b53d-64fc75b99a5f-kube-api-access-jrx2m\") pod \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " Oct 30 17:07:34.738321 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.719609 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f1c516-77d4-4e05-b53d-64fc75b99a5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f1c516-77d4-4e05-b53d-64fc75b99a5f-openshift-service-ca\") pod \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " Oct 30 17:07:34.738321 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.719695 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-metrics-certs\") pod \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " Oct 30 17:07:34.738321 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.719734 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-maas-api-tls\") pod \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\" (UID: \"85f1c516-77d4-4e05-b53d-64fc75b99a5f\") " Oct 30 17:07:34.738321 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.723228 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-maas-api-tls" pod "85f1c516-77d4-4e05-b53d-64fc75b99a5f" (UID: "85f1c516-77d4-4e05-b53d-64fc75b99a5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:34.742795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.741393 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f1c516-77d4-4e05-b53d-64fc75b99a5f-openshift-service-ca" pod "85f1c516-77d4-4e05-b53d-64fc75b99a5f" (UID: "85f1c516-77d4-4e05-b53d-64fc75b99a5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:34.783207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.776254 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f1c516-77d4-4e05-b53d-64fc75b99a5f-kube-api-access-jrx2m" pod "85f1c516-77d4-4e05-b53d-64fc75b99a5f" (UID: "85f1c516-77d4-4e05-b53d-64fc75b99a5f"). InnerVolumeSpecName "kube-api-access-jrx2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:34.783207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.777094 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-metrics-certs" pod "85f1c516-77d4-4e05-b53d-64fc75b99a5f" (UID: "85f1c516-77d4-4e05-b53d-64fc75b99a5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:34.825014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.824490 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrx2m\" (UniqueName: \"kubernetes.io/projected/85f1c516-77d4-4e05-b53d-64fc75b99a5f-kube-api-access-jrx2m\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:34.825014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.824540 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f1c516-77d4-4e05-b53d-64fc75b99a5f-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:34.825014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.824556 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:34.825014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:34.824607 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f1c516-77d4-4e05-b53d-64fc75b99a5f-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:35.020842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:35.020801 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:35.123386 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:35.123307 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" Oct 30 17:07:35.123386 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:35.123331 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4" event={"ID":"85f1c516-77d4-4e05-b53d-64fc75b99a5f","Type":"ContainerDied","Data":"2a6a97a43e2c6a7ae21db1ceb15734c63cd92130e769d5d1bfba4ba0b82980ce"} Oct 30 17:07:35.123656 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:35.123401 4828 scope.go:122] "RemoveContainer" containerID="e2f78e14dac51299e481469f0c1aa3fb60e73f76fe1ea283cc944d0cf9ee7954" Oct 30 17:07:35.180909 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:35.180856 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4"] Oct 30 17:07:35.193696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:35.193665 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4aba8b6b-585d5b5dff-qk9b4"] Oct 30 17:07:35.216945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:35.216735 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:35.350941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:35.348854 4828 patch_prober.go:29] interesting pod/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:07:35.363275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:35.362839 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc"] Oct 30 17:07:35.363275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:35.363099 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" podUID="6d021b80-bf6b-4392-a3e9-465536ff5fae" containerName="istio-proxy" containerID="cri-o://6be4b48064805f1165b55aae1abbfe1b71d88d6a6dd45569ba9f94b2c3edefe0" gracePeriod=30 Oct 30 17:07:35.370391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:35.363797 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" podUID="6d021b80-bf6b-4392-a3e9-465536ff5fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:35.397998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:35.397923 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:35.603983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:35.594383 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:36.495780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:36.495732 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f1c516-77d4-4e05-b53d-64fc75b99a5f" path="/var/lib/kubelet/pods/85f1c516-77d4-4e05-b53d-64fc75b99a5f/volumes" Oct 30 17:07:36.583829 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:36.583779 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:36.985129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:36.985094 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l"] Oct 30 17:07:36.986232 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:36.986204 4828 cpu_manager.go:414] "containerMap: removing container" podUID="85f1c516-77d4-4e05-b53d-64fc75b99a5f" containerName="maas-api" Oct 30 17:07:36.986232 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:36.986231 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f1c516-77d4-4e05-b53d-64fc75b99a5f" containerName="maas-api" Oct 30 17:07:36.986458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:36.986436 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f1c516-77d4-4e05-b53d-64fc75b99a5f" containerName="maas-api" Oct 30 17:07:36.992064 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:36.992037 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.009679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.009648 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c425a3e3-openshift-default-dockercfg-szdgz\"" Oct 30 17:07:37.036489 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.036454 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l"] Oct 30 17:07:37.088909 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.088869 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:37.095243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.095213 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-credential-socket\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.095406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.095383 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-socket\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.095475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.095452 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istiod-ca-cert\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.095582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.095562 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-podinfo\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.095668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.095642 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-certs\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.095712 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.095696 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-447c9\" (UniqueName: \"kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-kube-api-access-447c9\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.095821 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.095803 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-envoy\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.095881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.095866 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-data\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.095973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.095956 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-token\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.197036 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.196999 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-envoy\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.197210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.197146 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-data\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.197210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.197198 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-token\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.197288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.197255 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-credential-socket\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.197492 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.197470 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-socket\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.197632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.197582 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istiod-ca-cert\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.197632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.197624 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-podinfo\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.197752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.197675 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-credential-socket\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.197806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.197750 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-certs\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.197806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.197797 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447c9\" (UniqueName: \"kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-kube-api-access-447c9\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.197934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.197916 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-socket\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.198132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.198106 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-certs\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.198220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.198158 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-data\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.198399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.198370 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istiod-ca-cert\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.199671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.199647 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-envoy\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.200186 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.200167 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-podinfo\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.221816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.221785 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-token\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.222324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.222293 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-447c9\" (UniqueName: \"kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-kube-api-access-447c9\") pod \"e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.309929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.300529 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:37.321344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.320075 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:37.525945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:37.523477 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:37.974207 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:07:37.973110 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:07:38.141343 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:07:38.141306 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4bd50f_93c7_4ba8_8e58_ef48c1a178a4.slice/crio-99a43e5819b7d06b693187e289e13fb971bb3997bfe454301941337f061036f9 WatchSource:0}: Error finding container 99a43e5819b7d06b693187e289e13fb971bb3997bfe454301941337f061036f9: Status 404 returned error can't find the container with id 99a43e5819b7d06b693187e289e13fb971bb3997bfe454301941337f061036f9 Oct 30 17:07:38.144352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:38.144307 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:38.144466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:38.144402 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:38.144466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:38.144435 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:38.145037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:38.145010 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l"] Oct 30 17:07:38.162246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:38.162160 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" event={"ID":"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4","Type":"ContainerStarted","Data":"99a43e5819b7d06b693187e289e13fb971bb3997bfe454301941337f061036f9"} Oct 30 17:07:39.002146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.002113 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:07:39.098894 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.098810 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-token\" (UniqueName: \"kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-token\") pod \"6d021b80-bf6b-4392-a3e9-465536ff5fae\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " Oct 30 17:07:39.098894 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.098913 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-data\") pod \"6d021b80-bf6b-4392-a3e9-465536ff5fae\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " Oct 30 17:07:39.099169 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.098943 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-podinfo\") pod \"6d021b80-bf6b-4392-a3e9-465536ff5fae\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " Oct 30 17:07:39.099169 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.099022 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-certs\") pod \"6d021b80-bf6b-4392-a3e9-465536ff5fae\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " Oct 30 17:07:39.099294 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.099263 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-data" pod "6d021b80-bf6b-4392-a3e9-465536ff5fae" (UID: "6d021b80-bf6b-4392-a3e9-465536ff5fae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:39.099354 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.099292 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-certs" pod "6d021b80-bf6b-4392-a3e9-465536ff5fae" (UID: "6d021b80-bf6b-4392-a3e9-465536ff5fae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:39.099423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.099406 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-credential-socket\") pod \"6d021b80-bf6b-4392-a3e9-465536ff5fae\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " Oct 30 17:07:39.099499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.099483 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-envoy\") pod \"6d021b80-bf6b-4392-a3e9-465536ff5fae\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " Oct 30 17:07:39.099592 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.099578 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d021b80-bf6b-4392-a3e9-465536ff5fae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d021b80-bf6b-4392-a3e9-465536ff5fae-istiod-ca-cert\") pod \"6d021b80-bf6b-4392-a3e9-465536ff5fae\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " Oct 30 17:07:39.099769 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.099751 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-kube-api-access-5zw79\" (UniqueName: \"kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-kube-api-access-5zw79\") pod \"6d021b80-bf6b-4392-a3e9-465536ff5fae\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " Oct 30 17:07:39.099830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.099815 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-socket\") pod \"6d021b80-bf6b-4392-a3e9-465536ff5fae\" (UID: \"6d021b80-bf6b-4392-a3e9-465536ff5fae\") " Oct 30 17:07:39.100616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.100591 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:39.100673 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.100626 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:39.101148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.101091 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-credential-socket" pod "6d021b80-bf6b-4392-a3e9-465536ff5fae" (UID: "6d021b80-bf6b-4392-a3e9-465536ff5fae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:39.101233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.101175 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d021b80-bf6b-4392-a3e9-465536ff5fae-istiod-ca-cert" pod "6d021b80-bf6b-4392-a3e9-465536ff5fae" (UID: "6d021b80-bf6b-4392-a3e9-465536ff5fae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:39.101233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.101208 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-socket" pod "6d021b80-bf6b-4392-a3e9-465536ff5fae" (UID: "6d021b80-bf6b-4392-a3e9-465536ff5fae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:39.101737 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.101699 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-token" pod "6d021b80-bf6b-4392-a3e9-465536ff5fae" (UID: "6d021b80-bf6b-4392-a3e9-465536ff5fae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:39.102689 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.102663 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-podinfo" pod "6d021b80-bf6b-4392-a3e9-465536ff5fae" (UID: "6d021b80-bf6b-4392-a3e9-465536ff5fae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:07:39.103184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.103159 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-kube-api-access-5zw79" pod "6d021b80-bf6b-4392-a3e9-465536ff5fae" (UID: "6d021b80-bf6b-4392-a3e9-465536ff5fae"). InnerVolumeSpecName "kube-api-access-5zw79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:39.103275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.103233 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-envoy" pod "6d021b80-bf6b-4392-a3e9-465536ff5fae" (UID: "6d021b80-bf6b-4392-a3e9-465536ff5fae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:39.174675 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.174630 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" event={"ID":"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4","Type":"ContainerStarted","Data":"6c58eee8fedf51587c1839a7d8587d6fabf97d5aa33acab6e59983f0bfed6ce0"} Oct 30 17:07:39.176521 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.176490 4828 generic.go:356] "Generic (PLEG): container finished" podID="6d021b80-bf6b-4392-a3e9-465536ff5fae" containerID="6be4b48064805f1165b55aae1abbfe1b71d88d6a6dd45569ba9f94b2c3edefe0" exitCode=0 Oct 30 17:07:39.176932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.176548 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" Oct 30 17:07:39.176932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.176566 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" event={"ID":"6d021b80-bf6b-4392-a3e9-465536ff5fae","Type":"ContainerDied","Data":"6be4b48064805f1165b55aae1abbfe1b71d88d6a6dd45569ba9f94b2c3edefe0"} Oct 30 17:07:39.176932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.176620 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc" event={"ID":"6d021b80-bf6b-4392-a3e9-465536ff5fae","Type":"ContainerDied","Data":"80279e807a3d54f3111669070a28a502aa9cd1b62dd85d8f566faa03b70336b5"} Oct 30 17:07:39.176932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.176644 4828 scope.go:122] "RemoveContainer" containerID="6be4b48064805f1165b55aae1abbfe1b71d88d6a6dd45569ba9f94b2c3edefe0" Oct 30 17:07:39.189445 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.189421 4828 scope.go:122] "RemoveContainer" containerID="6be4b48064805f1165b55aae1abbfe1b71d88d6a6dd45569ba9f94b2c3edefe0" Oct 30 17:07:39.189842 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:07:39.189816 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be4b48064805f1165b55aae1abbfe1b71d88d6a6dd45569ba9f94b2c3edefe0\": container with ID starting with 6be4b48064805f1165b55aae1abbfe1b71d88d6a6dd45569ba9f94b2c3edefe0 not found: ID does not exist" containerID="6be4b48064805f1165b55aae1abbfe1b71d88d6a6dd45569ba9f94b2c3edefe0" Oct 30 17:07:39.189891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.189853 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be4b48064805f1165b55aae1abbfe1b71d88d6a6dd45569ba9f94b2c3edefe0"} err="failed to get container status \"6be4b48064805f1165b55aae1abbfe1b71d88d6a6dd45569ba9f94b2c3edefe0\": rpc error: code = NotFound desc = could not find container \"6be4b48064805f1165b55aae1abbfe1b71d88d6a6dd45569ba9f94b2c3edefe0\": container with ID starting with 6be4b48064805f1165b55aae1abbfe1b71d88d6a6dd45569ba9f94b2c3edefe0 not found: ID does not exist" Oct 30 17:07:39.202534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.202497 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d021b80-bf6b-4392-a3e9-465536ff5fae-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:39.202534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.202534 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zw79\" (UniqueName: \"kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-kube-api-access-5zw79\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:39.202730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.202551 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:39.202730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.202566 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:39.202730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.202580 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:39.202730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.202593 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:39.202730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.202605 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d021b80-bf6b-4392-a3e9-465536ff5fae-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:07:39.222882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.222824 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" podStartSLOduration=3.22280947 podStartE2EDuration="3.22280947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:39.220506327 +0000 UTC m=+1236.866016507" watchObservedRunningTime="2026-10-30 17:07:39.22280947 +0000 UTC m=+1236.868319649" Oct 30 17:07:39.261364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.261325 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc"] Oct 30 17:07:39.267954 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.267886 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-4aba8b6b-openshift-default-5c8665b485-qlmpc"] Oct 30 17:07:39.320963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.320892 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:39.326543 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.326517 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:39.611076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:39.609823 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:40.189851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:40.189813 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:40.190920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:40.190881 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:07:40.494965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:40.494833 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d021b80-bf6b-4392-a3e9-465536ff5fae" path="/var/lib/kubelet/pods/6d021b80-bf6b-4392-a3e9-465536ff5fae/volumes" Oct 30 17:07:40.881155 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:40.881114 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:41.638932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:41.634538 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:41.788376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:41.788301 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:42.788756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:42.788700 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:43.260204 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.260149 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:43.523978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.523856 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:43.618638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.618594 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5"] Oct 30 17:07:43.619432 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.619414 4828 cpu_manager.go:414] "containerMap: removing container" podUID="6d021b80-bf6b-4392-a3e9-465536ff5fae" containerName="istio-proxy" Oct 30 17:07:43.619511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.619434 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d021b80-bf6b-4392-a3e9-465536ff5fae" containerName="istio-proxy" Oct 30 17:07:43.619570 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.619560 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d021b80-bf6b-4392-a3e9-465536ff5fae" containerName="istio-proxy" Oct 30 17:07:43.623400 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.623377 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" Oct 30 17:07:43.643212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.643182 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5"] Oct 30 17:07:43.649695 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.649663 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c425a3e3\"" Oct 30 17:07:43.649940 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.649787 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c425a3e3-dockercfg-dzb6p\"" Oct 30 17:07:43.686829 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.686659 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:43.707856 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.707334 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr"] Oct 30 17:07:43.713101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.711390 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.734122 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.734091 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb"] Oct 30 17:07:43.737787 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.737476 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmplc\" (UniqueName: \"kubernetes.io/projected/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-kube-api-access-cmplc\") pod \"payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5\" (UID: \"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" Oct 30 17:07:43.737787 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.737625 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5\" (UID: \"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" Oct 30 17:07:43.739095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.739075 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" Oct 30 17:07:43.739323 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.739096 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6d4f99-openshift-default-dockercfg-chllj\"" Oct 30 17:07:43.742661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.742508 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr"] Oct 30 17:07:43.761081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.761023 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb"] Oct 30 17:07:43.840108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.840022 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.840108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.840078 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5\" (UID: \"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" Oct 30 17:07:43.840625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.840213 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xcvs\" (UniqueName: \"kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-kube-api-access-9xcvs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.840625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.840310 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-token\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.840625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.840364 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.840625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.840573 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-envoy\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.840845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.840685 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfk7k\" (UniqueName: \"kubernetes.io/projected/77d5e34f-129b-48a3-9aa3-e8439e81b6db-kube-api-access-hfk7k\") pod \"payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb\" (UID: \"77d5e34f-129b-48a3-9aa3-e8439e81b6db\") " pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" Oct 30 17:07:43.840845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.840739 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-credential-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.840845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.840791 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-certs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.840991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.840894 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-data\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.840991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.840939 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5\" (UID: \"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" Oct 30 17:07:43.840991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.840945 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-podinfo\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.841107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.841035 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmplc\" (UniqueName: \"kubernetes.io/projected/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-kube-api-access-cmplc\") pod \"payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5\" (UID: \"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" Oct 30 17:07:43.841150 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.841117 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d5e34f-129b-48a3-9aa3-e8439e81b6db-plugins-config-volume\") pod \"payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb\" (UID: \"77d5e34f-129b-48a3-9aa3-e8439e81b6db\") " pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" Oct 30 17:07:43.841615 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.841592 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:43.865249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.865216 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmplc\" (UniqueName: \"kubernetes.io/projected/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-kube-api-access-cmplc\") pod \"payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5\" (UID: \"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" Oct 30 17:07:43.940702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.940659 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" Oct 30 17:07:43.948929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.944070 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xcvs\" (UniqueName: \"kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-kube-api-access-9xcvs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.948929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.944170 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-token\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.948929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.944218 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.948929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.944330 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-envoy\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.948929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.944389 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfk7k\" (UniqueName: \"kubernetes.io/projected/77d5e34f-129b-48a3-9aa3-e8439e81b6db-kube-api-access-hfk7k\") pod \"payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb\" (UID: \"77d5e34f-129b-48a3-9aa3-e8439e81b6db\") " pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" Oct 30 17:07:43.948929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.944441 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-credential-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.948929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.944494 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-certs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.948929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.944548 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-data\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.948929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.944573 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-podinfo\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.948929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.944670 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d5e34f-129b-48a3-9aa3-e8439e81b6db-plugins-config-volume\") pod \"payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb\" (UID: \"77d5e34f-129b-48a3-9aa3-e8439e81b6db\") " pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" Oct 30 17:07:43.948929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.944710 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.951980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.951936 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.953072 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.953034 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.958783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.958099 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-envoy\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.958783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.958672 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-credential-socket\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.959411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.959314 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-certs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:43.960588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.960527 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d5e34f-129b-48a3-9aa3-e8439e81b6db-plugins-config-volume\") pod \"payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb\" (UID: \"77d5e34f-129b-48a3-9aa3-e8439e81b6db\") " pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" Oct 30 17:07:43.986524 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:43.961080 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-data\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:44.005989 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.005956 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-podinfo\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:44.030398 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.030361 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-token\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:44.031407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.031342 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xcvs\" (UniqueName: \"kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-kube-api-access-9xcvs\") pod \"e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:44.044095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.042293 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:44.090974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.083762 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfk7k\" (UniqueName: \"kubernetes.io/projected/77d5e34f-129b-48a3-9aa3-e8439e81b6db-kube-api-access-hfk7k\") pod \"payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb\" (UID: \"77d5e34f-129b-48a3-9aa3-e8439e81b6db\") " pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" Oct 30 17:07:44.090974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.084410 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:44.164922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.164148 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" Oct 30 17:07:44.189929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.188186 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" Oct 30 17:07:44.423677 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.399512 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" Oct 30 17:07:44.869789 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:07:44.869741 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a5cf9c_0f32_4a03_9d5c_2a2764bbc47d.slice/crio-68afacca90c3161d47290ef192870d799d6e6eed37fa95b039343e86bda502f9 WatchSource:0}: Error finding container 68afacca90c3161d47290ef192870d799d6e6eed37fa95b039343e86bda502f9: Status 404 returned error can't find the container with id 68afacca90c3161d47290ef192870d799d6e6eed37fa95b039343e86bda502f9 Oct 30 17:07:44.871259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.871171 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5"] Oct 30 17:07:44.879556 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.879520 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr"] Oct 30 17:07:44.890421 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:07:44.890136 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3f407f_bf79_4e39_8c17_c1e2a33eadae.slice/crio-001102abcc95e038cb7249f31d610c6bb165eb6e9f1381eee58f976f396505f9 WatchSource:0}: Error finding container 001102abcc95e038cb7249f31d610c6bb165eb6e9f1381eee58f976f396505f9: Status 404 returned error can't find the container with id 001102abcc95e038cb7249f31d610c6bb165eb6e9f1381eee58f976f396505f9 Oct 30 17:07:44.913219 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:07:44.913143 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d5e34f_129b_48a3_9aa3_e8439e81b6db.slice/crio-8be46d09136005473260051e99dc6b7fe4868e71b5d1367e76e6a47d9356403e WatchSource:0}: Error finding container 8be46d09136005473260051e99dc6b7fe4868e71b5d1367e76e6a47d9356403e: Status 404 returned error can't find the container with id 8be46d09136005473260051e99dc6b7fe4868e71b5d1367e76e6a47d9356403e Oct 30 17:07:44.916913 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.916868 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb"] Oct 30 17:07:44.919093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.918351 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:44.919093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.918433 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:44.919093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:44.918482 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:45.531495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:45.531457 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" event={"ID":"5f3f407f-bf79-4e39-8c17-c1e2a33eadae","Type":"ContainerStarted","Data":"e2a2a6d091bbc13b550b409f779e3e3234b92a3827254279450b685a2e618cd0"} Oct 30 17:07:45.531725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:45.531522 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" event={"ID":"5f3f407f-bf79-4e39-8c17-c1e2a33eadae","Type":"ContainerStarted","Data":"001102abcc95e038cb7249f31d610c6bb165eb6e9f1381eee58f976f396505f9"} Oct 30 17:07:45.533295 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:45.533263 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" event={"ID":"77d5e34f-129b-48a3-9aa3-e8439e81b6db","Type":"ContainerStarted","Data":"a2afb8eeb4798e831dacc3003ed8bdad84c49f0698c3117c2f711bd2b7668f6d"} Oct 30 17:07:45.533415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:45.533323 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" Oct 30 17:07:45.533415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:45.533351 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" event={"ID":"77d5e34f-129b-48a3-9aa3-e8439e81b6db","Type":"ContainerStarted","Data":"8be46d09136005473260051e99dc6b7fe4868e71b5d1367e76e6a47d9356403e"} Oct 30 17:07:45.534990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:45.534967 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" event={"ID":"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d","Type":"ContainerStarted","Data":"564fec47d6be2160f4d977e2675b149ed465e6caba81f3e49d71f953dc01645f"} Oct 30 17:07:45.535087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:45.535006 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" event={"ID":"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d","Type":"ContainerStarted","Data":"68afacca90c3161d47290ef192870d799d6e6eed37fa95b039343e86bda502f9"} Oct 30 17:07:45.535087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:45.535031 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" Oct 30 17:07:45.583057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:45.582994 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" podStartSLOduration=2.5829719730000003 podStartE2EDuration="2.582971973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:45.579091537 +0000 UTC m=+1243.224601717" watchObservedRunningTime="2026-10-30 17:07:45.582971973 +0000 UTC m=+1243.228482154" Oct 30 17:07:45.623713 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:45.623655 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" podStartSLOduration=2.623639175 podStartE2EDuration="2.623639175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:45.620559897 +0000 UTC m=+1243.266070080" watchObservedRunningTime="2026-10-30 17:07:45.623639175 +0000 UTC m=+1243.269149355" Oct 30 17:07:45.665800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:45.665747 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" podStartSLOduration=2.665734312 podStartE2EDuration="2.665734312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:45.662306717 +0000 UTC m=+1243.307816899" watchObservedRunningTime="2026-10-30 17:07:45.665734312 +0000 UTC m=+1243.311244492" Oct 30 17:07:45.992971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:45.992834 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:46.043786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:46.043744 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:46.049743 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:46.049713 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:46.542583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:46.542546 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:46.543525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:46.543500 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:07:48.152192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:48.152150 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:48.345924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:48.328886 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:50.086924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.072952 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8"] Oct 30 17:07:50.086924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.079743 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:50.095444 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.095412 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.116971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.116884 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p"] Oct 30 17:07:50.165079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.163543 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.165079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.163609 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.165079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.163785 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.165079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.163830 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.165079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.163872 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.165079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.164036 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.165079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.164075 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.165079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.164126 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.165079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.164270 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gkqg\" (UniqueName: \"kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-kube-api-access-2gkqg\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.360104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.359981 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.360388 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.360360 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.360602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.360581 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.360746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.360728 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.360883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.360866 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.361082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.361063 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.363278 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.361406 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gkqg\" (UniqueName: \"kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-kube-api-access-2gkqg\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.363278 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.361511 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.363278 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.361605 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.363278 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.362362 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.507956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.460863 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:50.507956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.478735 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.507956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.479031 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.507956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.479425 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.507956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.480495 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.604024 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.582583 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8"] Oct 30 17:07:50.604024 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.582931 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.755535 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.755445 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.755877 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.755841 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.756064 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.756043 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.756269 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.756253 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.756439 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.756409 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.756677 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.756659 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fj7b\" (UniqueName: \"kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-kube-api-access-5fj7b\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.756930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.756910 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04e6e28-2061-4325-ae70-980c86ae72e8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.757132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.757113 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.757334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.757299 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.862741 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.862703 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.864291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.863872 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.864556 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.864526 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.865131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.865004 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.865354 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.865329 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.866175 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.866155 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.867985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.867960 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fj7b\" (UniqueName: \"kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-kube-api-access-5fj7b\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.869226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.869203 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04e6e28-2061-4325-ae70-980c86ae72e8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.869397 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.869355 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.869683 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.869654 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.876402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.870041 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.959029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.936659 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p"] Oct 30 17:07:50.959353 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.937247 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:50.959499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.947752 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.959647 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.949938 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04e6e28-2061-4325-ae70-980c86ae72e8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.959750 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.950175 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:50.959855 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:50.958414 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:51.016927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:51.006829 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gkqg\" (UniqueName: \"kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-kube-api-access-2gkqg\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:51.045023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:51.034669 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:51.143985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:51.090648 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-n8mh8\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:51.143985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:51.094934 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fj7b\" (UniqueName: \"kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-kube-api-access-5fj7b\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:51.161799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:51.161729 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:51.186991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:51.170106 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:07:51.187322 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:51.178165 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6j56p\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:51.371776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:51.371743 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:51.564423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:51.564313 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:51.719981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:51.714138 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:51.765221 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:51.765132 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:52.417349 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:07:52.417313 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ba7cfe_7c22_4bf6_97f8_ce90c9269462.slice/crio-f653738a5e05841b6a3beb157e81e3f5e5eefffad30516aca0b01ca1e74147de WatchSource:0}: Error finding container f653738a5e05841b6a3beb157e81e3f5e5eefffad30516aca0b01ca1e74147de: Status 404 returned error can't find the container with id f653738a5e05841b6a3beb157e81e3f5e5eefffad30516aca0b01ca1e74147de Oct 30 17:07:52.430594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:52.430012 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:52.430999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:52.430950 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:52.431408 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:52.431383 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:52.438695 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:52.438666 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8"] Oct 30 17:07:52.445695 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:52.445499 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p"] Oct 30 17:07:52.446826 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:52.446796 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:52.446953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:52.446855 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:52.446953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:52.446922 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:07:52.917635 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:52.917598 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" event={"ID":"a04e6e28-2061-4325-ae70-980c86ae72e8","Type":"ContainerStarted","Data":"a039b243aa94e9460182d4eeddd0a396c1c12a3f6ad8ac66b8698fb35bf639df"} Oct 30 17:07:52.918337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:52.918306 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" event={"ID":"a04e6e28-2061-4325-ae70-980c86ae72e8","Type":"ContainerStarted","Data":"147fc0bc45051cb0b36a0ca2a85316b536ba7fe6c56f659967f812445379e5ab"} Oct 30 17:07:52.920234 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:52.920203 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" event={"ID":"61ba7cfe-7c22-4bf6-97f8-ce90c9269462","Type":"ContainerStarted","Data":"fa3d01ee6f657022887cfd7e37e34388db433fb66a84cf043d89e24ebcf40a45"} Oct 30 17:07:52.920361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:52.920255 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" event={"ID":"61ba7cfe-7c22-4bf6-97f8-ce90c9269462","Type":"ContainerStarted","Data":"f653738a5e05841b6a3beb157e81e3f5e5eefffad30516aca0b01ca1e74147de"} Oct 30 17:07:53.080519 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:53.080428 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" podStartSLOduration=4.080409188 podStartE2EDuration="4.080409188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:52.973292623 +0000 UTC m=+1250.618802840" watchObservedRunningTime="2026-10-30 17:07:53.080409188 +0000 UTC m=+1250.725919371" Oct 30 17:07:53.099124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:53.099062 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" podStartSLOduration=4.099041241 podStartE2EDuration="4.099041241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:53.035657669 +0000 UTC m=+1250.681167850" watchObservedRunningTime="2026-10-30 17:07:53.099041241 +0000 UTC m=+1250.744551424" Oct 30 17:07:53.418098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:53.418000 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:07:53.425459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:53.425430 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n8mh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:07:53.425640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:53.425489 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:07:53.744107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:53.744014 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:07:53.749036 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:53.749001 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6j56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:07:53.749300 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:53.749265 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:07:53.920327 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:53.920286 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:54.376259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:54.376220 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n8mh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:07:54.376557 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:54.376526 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:07:54.737985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:54.737240 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6j56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:07:54.737985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:54.737305 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:07:55.387939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:55.377184 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n8mh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:07:55.387939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:55.377260 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:07:55.769921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:55.764613 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6j56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:07:55.769921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:55.764669 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:07:56.026893 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:56.026677 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:56.389825 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:56.389665 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n8mh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:07:56.390194 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:56.390162 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:07:56.606129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:56.606101 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" Oct 30 17:07:56.610222 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:56.610188 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" Oct 30 17:07:56.748656 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:56.748532 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6j56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:07:56.748936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:56.748887 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:07:56.754824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:56.754609 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:57.379386 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:57.379337 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n8mh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:07:57.380012 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:57.379977 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:07:57.734024 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:57.733881 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6j56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:07:57.734776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:57.734271 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:07:58.380747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:58.380709 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n8mh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:07:58.381403 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:58.381369 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:07:58.733423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:58.733328 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6j56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:07:58.733734 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:58.733691 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:07:58.958926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:58.949881 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:59.133456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:59.133414 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:59.379103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:59.379065 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n8mh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:07:59.379395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:59.379365 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:07:59.421990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:59.396736 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:07:59.736977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:59.735320 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6j56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:07:59.736977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:59.735383 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:07:59.896797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:07:59.893826 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:00.055838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:00.055790 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:00.401094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:00.401011 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n8mh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:08:00.427730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:00.427690 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:08:00.733176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:00.733091 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6j56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:08:00.733176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:00.733160 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:08:01.156925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:01.156055 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:01.376280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:01.376241 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n8mh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:08:01.376511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:01.376491 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:08:01.733756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:01.733707 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6j56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 17:08:01.734233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:01.733786 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 17:08:02.376922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:02.376859 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-n8mh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 17:08:02.377219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:02.377188 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 17:08:03.279212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:03.279175 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:03.448231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:03.448192 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:03.741625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:03.740993 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6j56p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:08:03.741625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:03.741071 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:08:03.979931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:03.972971 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:08:03.979931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:03.974828 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:08:03.979931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:03.976036 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:08:04.152131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:04.152095 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:08:04.155033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:04.154999 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:08:04.156048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:04.156029 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:08:05.107576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.107537 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:05.258366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.258322 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:05.305924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.304834 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz"] Oct 30 17:08:05.314275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.313491 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" Oct 30 17:08:05.334187 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.334161 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz"] Oct 30 17:08:05.341322 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.341153 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-a43843\"" Oct 30 17:08:05.341322 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.341230 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-a43843-dockercfg-bqdqs\"" Oct 30 17:08:05.503981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.496197 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:05.504276 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.501524 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9c5027-e57b-4b81-bdaf-3a8910099773-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz\" (UID: \"3d9c5027-e57b-4b81-bdaf-3a8910099773\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" Oct 30 17:08:05.514044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.510688 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7f9j\" (UniqueName: \"kubernetes.io/projected/3d9c5027-e57b-4b81-bdaf-3a8910099773-kube-api-access-m7f9j\") pod \"payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz\" (UID: \"3d9c5027-e57b-4b81-bdaf-3a8910099773\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" Oct 30 17:08:05.622794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.622745 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9c5027-e57b-4b81-bdaf-3a8910099773-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz\" (UID: \"3d9c5027-e57b-4b81-bdaf-3a8910099773\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" Oct 30 17:08:05.623174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.623132 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7f9j\" (UniqueName: \"kubernetes.io/projected/3d9c5027-e57b-4b81-bdaf-3a8910099773-kube-api-access-m7f9j\") pod \"payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz\" (UID: \"3d9c5027-e57b-4b81-bdaf-3a8910099773\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" Oct 30 17:08:05.859922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.857024 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9c5027-e57b-4b81-bdaf-3a8910099773-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz\" (UID: \"3d9c5027-e57b-4b81-bdaf-3a8910099773\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" Oct 30 17:08:05.887348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.887314 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7f9j\" (UniqueName: \"kubernetes.io/projected/3d9c5027-e57b-4b81-bdaf-3a8910099773-kube-api-access-m7f9j\") pod \"payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz\" (UID: \"3d9c5027-e57b-4b81-bdaf-3a8910099773\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" Oct 30 17:08:05.938320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:05.938282 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" Oct 30 17:08:06.252484 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:08:06.252455 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9c5027_e57b_4b81_bdaf_3a8910099773.slice/crio-c62ab46e64c9298d32b9c64e84c97d13098d80a1c468f4d4a4b70468c168ab7f WatchSource:0}: Error finding container c62ab46e64c9298d32b9c64e84c97d13098d80a1c468f4d4a4b70468c168ab7f: Status 404 returned error can't find the container with id c62ab46e64c9298d32b9c64e84c97d13098d80a1c468f4d4a4b70468c168ab7f Oct 30 17:08:06.256725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:06.256696 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz"] Oct 30 17:08:06.863615 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:06.863582 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" event={"ID":"3d9c5027-e57b-4b81-bdaf-3a8910099773","Type":"ContainerStarted","Data":"0c4ffa6bddb053459d3b3cd9702fe2f99bd2460e1ddbb2c1d03a3d68725c4f88"} Oct 30 17:08:06.863812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:06.863629 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" event={"ID":"3d9c5027-e57b-4b81-bdaf-3a8910099773","Type":"ContainerStarted","Data":"c62ab46e64c9298d32b9c64e84c97d13098d80a1c468f4d4a4b70468c168ab7f"} Oct 30 17:08:06.863877 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:06.863830 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" Oct 30 17:08:06.908057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:06.907998 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" podStartSLOduration=1.9079805319999998 podStartE2EDuration="1.907980532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:06.906546552 +0000 UTC m=+1264.552056731" watchObservedRunningTime="2026-10-30 17:08:06.907980532 +0000 UTC m=+1264.553490711" Oct 30 17:08:07.329067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.329022 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc"] Oct 30 17:08:07.334443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.334417 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.357597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.357569 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c425a3e3-serving-cert\"" Oct 30 17:08:07.358307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.358281 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc"] Oct 30 17:08:07.395565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.395523 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-metrics-certs\") pod \"maas-api-e2e-mt-c425a3e3-5df9746659-j66pc\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.395741 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.395586 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2208a20-2b4b-4cbd-a471-c39873e21f09-openshift-service-ca\") pod \"maas-api-e2e-mt-c425a3e3-5df9746659-j66pc\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.395788 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.395744 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-maas-api-tls\") pod \"maas-api-e2e-mt-c425a3e3-5df9746659-j66pc\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.396020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.395996 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22tjl\" (UniqueName: \"kubernetes.io/projected/d2208a20-2b4b-4cbd-a471-c39873e21f09-kube-api-access-22tjl\") pod \"maas-api-e2e-mt-c425a3e3-5df9746659-j66pc\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.453095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.453048 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:07.497494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.497460 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22tjl\" (UniqueName: \"kubernetes.io/projected/d2208a20-2b4b-4cbd-a471-c39873e21f09-kube-api-access-22tjl\") pod \"maas-api-e2e-mt-c425a3e3-5df9746659-j66pc\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.497681 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.497576 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-metrics-certs\") pod \"maas-api-e2e-mt-c425a3e3-5df9746659-j66pc\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.497681 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.497601 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2208a20-2b4b-4cbd-a471-c39873e21f09-openshift-service-ca\") pod \"maas-api-e2e-mt-c425a3e3-5df9746659-j66pc\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.497812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.497695 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-maas-api-tls\") pod \"maas-api-e2e-mt-c425a3e3-5df9746659-j66pc\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.498409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.498384 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2208a20-2b4b-4cbd-a471-c39873e21f09-openshift-service-ca\") pod \"maas-api-e2e-mt-c425a3e3-5df9746659-j66pc\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.500095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.500072 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-metrics-certs\") pod \"maas-api-e2e-mt-c425a3e3-5df9746659-j66pc\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.500245 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.500227 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-maas-api-tls\") pod \"maas-api-e2e-mt-c425a3e3-5df9746659-j66pc\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.522445 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.522414 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22tjl\" (UniqueName: \"kubernetes.io/projected/d2208a20-2b4b-4cbd-a471-c39873e21f09-kube-api-access-22tjl\") pod \"maas-api-e2e-mt-c425a3e3-5df9746659-j66pc\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.620403 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.619892 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:07.653877 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:07.651038 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:07.942229 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:08:07.942141 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:08:08.086058 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:08:08.086025 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2208a20_2b4b_4cbd_a471_c39873e21f09.slice/crio-93b090036763424cb2aefbc71cb458bedc947c9dc1ef325c362e69cc3d897a61 WatchSource:0}: Error finding container 93b090036763424cb2aefbc71cb458bedc947c9dc1ef325c362e69cc3d897a61: Status 404 returned error can't find the container with id 93b090036763424cb2aefbc71cb458bedc947c9dc1ef325c362e69cc3d897a61 Oct 30 17:08:08.087486 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:08.087460 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc"] Oct 30 17:08:08.989925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:08.985107 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" event={"ID":"d2208a20-2b4b-4cbd-a471-c39873e21f09","Type":"ContainerStarted","Data":"7ac013e39ee79b687edb7c5d0c53cc97929c08791191aaa6928a92d7a5ed746f"} Oct 30 17:08:08.989925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:08.985272 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" event={"ID":"d2208a20-2b4b-4cbd-a471-c39873e21f09","Type":"ContainerStarted","Data":"93b090036763424cb2aefbc71cb458bedc947c9dc1ef325c362e69cc3d897a61"} Oct 30 17:08:08.990499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:08.990140 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:09.041810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:09.041756 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" podStartSLOduration=1.7152034390000002 podStartE2EDuration="2.041741659s" totalImagesPullingTime="326.53822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:09.035562525 +0000 UTC m=+1266.681072707" watchObservedRunningTime="2026-10-30 17:08:09.041741659 +0000 UTC m=+1266.687251838" Oct 30 17:08:09.396131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:09.394413 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:11.155520 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:11.155485 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:12.956924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:12.953201 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:14.425205 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:14.425169 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:14.611493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:14.611088 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:14.783311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:14.783271 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:15.000185 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:15.000153 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:08:16.518922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:16.513002 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:17.880059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:17.880024 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" Oct 30 17:08:18.284717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:18.284680 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:18.516616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:18.516579 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:18.670238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:18.670141 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:18.863187 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:18.863152 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:20.000463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:19.992953 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:21.842267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:21.842228 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:23.655790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:23.655752 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:25.523453 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:25.520419 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:26.218014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:26.217974 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:26.370132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:26.370097 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:26.878600 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:26.878564 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:27.097923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:27.092457 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:27.928866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:27.928832 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n"] Oct 30 17:08:27.935533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:27.935507 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:27.954966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:27.954935 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-a43843-serving-cert\"" Oct 30 17:08:27.957487 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:27.957461 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n"] Oct 30 17:08:28.016959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.016924 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e851e1-0779-48c0-9e4b-2236addceb52-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:28.017146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.016971 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:28.017146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.017118 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdjs\" (UniqueName: \"kubernetes.io/projected/21e851e1-0779-48c0-9e4b-2236addceb52-kube-api-access-dmdjs\") pod \"maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:28.017371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.017354 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:28.027106 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.027070 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:28.119352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.119315 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e851e1-0779-48c0-9e4b-2236addceb52-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:28.119531 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.119361 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:28.119531 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.119462 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdjs\" (UniqueName: \"kubernetes.io/projected/21e851e1-0779-48c0-9e4b-2236addceb52-kube-api-access-dmdjs\") pod \"maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:28.119642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.119631 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:28.120477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.120451 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e851e1-0779-48c0-9e4b-2236addceb52-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:28.122215 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.122191 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:28.122365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.122236 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:28.144738 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.144706 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdjs\" (UniqueName: \"kubernetes.io/projected/21e851e1-0779-48c0-9e4b-2236addceb52-kube-api-access-dmdjs\") pod \"maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:28.290979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.256012 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:28.362811 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.357618 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:28.595640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:28.595551 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:29.017019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:29.016981 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n"] Oct 30 17:08:29.642936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:29.636146 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" event={"ID":"21e851e1-0779-48c0-9e4b-2236addceb52","Type":"ContainerStarted","Data":"0514b2927385d0beeee280381f20746988e6bfdd291efcbbf4eb8672a7d1f824"} Oct 30 17:08:30.568932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:30.564301 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:30.689131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:30.687734 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" event={"ID":"21e851e1-0779-48c0-9e4b-2236addceb52","Type":"ContainerStarted","Data":"83b7f91715d3995db6fd63d6b6578410638caf8d4c92ce8c95ba1636d89e67b7"} Oct 30 17:08:30.689131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:30.688970 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:30.749211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:30.743875 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" podStartSLOduration=3.279238297 podStartE2EDuration="3.74385458s" totalImagesPullingTime="464.616283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:30.742362928 +0000 UTC m=+1288.387873108" watchObservedRunningTime="2026-10-30 17:08:30.74385458 +0000 UTC m=+1288.389364762" Oct 30 17:08:32.417242 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:32.417002 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:34.014925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:34.011443 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:35.992638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:35.992596 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:36.816633 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:36.816597 4828 generic.go:356] "Generic (PLEG): container finished" podID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerID="6c58eee8fedf51587c1839a7d8587d6fabf97d5aa33acab6e59983f0bfed6ce0" exitCode=0 Oct 30 17:08:36.816841 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:36.816676 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" event={"ID":"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4","Type":"ContainerDied","Data":"6c58eee8fedf51587c1839a7d8587d6fabf97d5aa33acab6e59983f0bfed6ce0"} Oct 30 17:08:36.817254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:36.817231 4828 scope.go:122] "RemoveContainer" containerID="6c58eee8fedf51587c1839a7d8587d6fabf97d5aa33acab6e59983f0bfed6ce0" Oct 30 17:08:36.819003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:36.818975 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:08:36.819111 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:36.819059 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:08:36.819175 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:36.819110 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:08:37.739759 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:37.739730 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:08:37.744546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:37.744513 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:37.844546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:37.844496 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" event={"ID":"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4","Type":"ContainerStarted","Data":"fcf78c4702e30a53e77433daa0aa97f7cb82ead72aaa47692a29f8bef5c4b477"} Oct 30 17:08:37.934227 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:37.934132 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:38.130160 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:38.130126 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:38.174429 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:08:38.174389 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:08:38.321072 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:38.321031 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:08:38.322843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:38.322814 4828 patch_prober.go:29] interesting pod/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 17:08:38.322992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:38.322869 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 17:08:38.816799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:38.816557 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:38.967024 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:38.965883 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:39.321011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:39.320959 4828 patch_prober.go:29] interesting pod/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 17:08:39.321227 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:39.321029 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 17:08:39.523029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:39.522977 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:39.898925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:39.898590 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:40.191092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:40.190999 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:08:40.320995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:40.320958 4828 patch_prober.go:29] interesting pod/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 17:08:40.321181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:40.321027 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 17:08:41.321399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:41.321362 4828 patch_prober.go:29] interesting pod/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 17:08:41.322052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:41.321442 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 17:08:41.694918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:41.694786 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:42.324346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:42.324305 4828 patch_prober.go:29] interesting pod/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 17:08:42.324746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:42.324388 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 17:08:43.330346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:43.330304 4828 patch_prober.go:29] interesting pod/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 17:08:43.330958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:43.330925 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 17:08:43.707271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:43.707172 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:44.324975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:44.324930 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:44.379328 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:44.379295 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:08:44.390523 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:44.390450 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:08:45.606883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:45.606846 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:45.816923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:45.800785 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:46.127554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:46.127507 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:46.305927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:46.302677 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:46.662923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:46.658531 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:47.596925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:47.589876 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:49.469826 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:49.469754 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:49.709499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:49.707043 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:49.985524 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:49.985481 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:50.147749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:50.147708 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:51.335921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:51.335139 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:51.762694 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:51.762661 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:51.984923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:51.972794 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:53.177797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:53.177753 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:54.987992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:54.987647 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:55.805166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:55.805129 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:56.284058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:56.284007 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:56.533925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:56.520423 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:56.999921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:56.996088 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:58.984171 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:58.984118 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:59.428626 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:59.428578 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:08:59.638670 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:08:59.638623 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:00.792925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:00.792266 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:01.346026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:01.336358 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:01.755087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:01.754968 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:01.921707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:01.921666 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:03.664660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:03.664623 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:05.846891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:05.846814 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:09:06.042354 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:06.042285 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:06.249099 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:06.249004 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:06.446678 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:06.446634 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:06.633970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:06.620117 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:06.851607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:06.851568 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:07.937127 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:09:07.937091 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:09:08.231460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:08.231374 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:08.390020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:08.389971 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:09.452577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.448283 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv"] Oct 30 17:09:09.457205 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.457179 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.483543 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.481059 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-728ad7-openshift-default-dockercfg-hsckd\"" Oct 30 17:09:09.483543 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.483428 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:09.522922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.522867 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv"] Oct 30 17:09:09.553119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.551918 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-credential-socket\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.553711 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.553570 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brj79\" (UniqueName: \"kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-kube-api-access-brj79\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.553970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.553947 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-certs\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.554253 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.554222 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3717264a-0933-41fa-9082-221ffa0c6ba5-istiod-ca-cert\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.554401 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.554383 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-envoy\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.554539 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.554520 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-podinfo\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.554722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.554704 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-data\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.554936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.554917 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-token\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.555116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.555098 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-socket\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.691915 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.691467 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-podinfo\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.712976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.711060 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-data\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.712976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.711133 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-token\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.712976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.711204 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-socket\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.712976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.711541 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-credential-socket\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.712976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.711617 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brj79\" (UniqueName: \"kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-kube-api-access-brj79\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.712976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.711670 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-certs\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.712976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.711741 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3717264a-0933-41fa-9082-221ffa0c6ba5-istiod-ca-cert\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.712976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.711770 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-envoy\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.724120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.720713 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-credential-socket\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.724120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.721160 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-data\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.724120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.721162 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-envoy\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.724120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.721479 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-socket\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.724120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.721735 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-certs\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.724120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.722668 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3717264a-0933-41fa-9082-221ffa0c6ba5-istiod-ca-cert\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.838629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.837951 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-podinfo\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.838629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.838455 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:09.847431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.847397 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-token\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:09.862069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:09.862032 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brj79\" (UniqueName: \"kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-kube-api-access-brj79\") pod \"e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:10.019053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:10.005577 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:10.098302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:10.098236 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:10.493464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:10.491803 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:09:10.493464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:10.492866 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:09:10.493464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:10.492968 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:09:10.512920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:10.512875 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv"] Oct 30 17:09:11.297565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:11.297516 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" event={"ID":"3717264a-0933-41fa-9082-221ffa0c6ba5","Type":"ContainerStarted","Data":"861575c602f265de08be93b415495370836f5d8563d0b29fa688e288e560e1cc"} Oct 30 17:09:11.297752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:11.297598 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" event={"ID":"3717264a-0933-41fa-9082-221ffa0c6ba5","Type":"ContainerStarted","Data":"f4a529f39a48b841735c694b3447678b608580211447e7c167a331ab412559ad"} Oct 30 17:09:11.363659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:11.363590 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" podStartSLOduration=2.363572214 podStartE2EDuration="2.363572214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:11.358927208 +0000 UTC m=+1329.004437379" watchObservedRunningTime="2026-10-30 17:09:11.363572214 +0000 UTC m=+1329.009082464" Oct 30 17:09:11.715011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:11.714922 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:12.113767 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:12.113736 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:12.119849 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:12.119817 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:12.311056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:12.311018 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:12.312358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:12.312331 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:09:13.377935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:13.370581 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:13.811137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:13.811088 4828 patch_prober.go:29] interesting pod/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:09:13.811351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:13.811153 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:09:13.890208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:13.890170 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:14.068804 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:14.068728 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:15.227023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:15.226820 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:15.400775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:15.399215 4828 generic.go:356] "Generic (PLEG): container finished" podID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerID="e2ca8adfbdccf2e68d612c0af1c36e56d7770ab912d1d8bcc4e57af7ea3b65ad" exitCode=0 Oct 30 17:09:15.400775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:15.399454 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" event={"ID":"836c4dab-6052-49e4-b60e-98bfa0f48b5b","Type":"ContainerDied","Data":"e2ca8adfbdccf2e68d612c0af1c36e56d7770ab912d1d8bcc4e57af7ea3b65ad"} Oct 30 17:09:15.400775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:15.400121 4828 scope.go:122] "RemoveContainer" containerID="e2ca8adfbdccf2e68d612c0af1c36e56d7770ab912d1d8bcc4e57af7ea3b65ad" Oct 30 17:09:15.402501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:15.402468 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:09:15.402608 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:15.402547 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:09:15.402608 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:15.402592 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:09:15.686207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:15.686170 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:15.856039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:15.855970 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:16.640633 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:16.639273 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" event={"ID":"836c4dab-6052-49e4-b60e-98bfa0f48b5b","Type":"ContainerStarted","Data":"c8b4a75c8b026693d7ea4d1015f4fb19e8cccbe8a314552e3d772411ac37c899"} Oct 30 17:09:17.298751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:17.298714 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:17.595982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:17.595038 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:09:17.602239 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:17.602205 4828 patch_prober.go:29] interesting pod/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 17:09:17.602532 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:17.602488 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 17:09:18.592356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:18.592312 4828 patch_prober.go:29] interesting pod/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 17:09:18.593019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:18.592985 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 17:09:18.835775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:18.835725 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2"] Oct 30 17:09:18.841200 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:18.841171 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:18.860487 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:18.860350 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2"] Oct 30 17:09:18.867277 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:18.867247 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7d1e5c-serving-cert\"" Oct 30 17:09:18.912089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:18.912049 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-metrics-certs\") pod \"maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:18.912400 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:18.912379 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6b648f-9f86-4064-9890-8b653d7539b8-openshift-service-ca\") pod \"maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:18.912709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:18.912689 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-maas-api-tls\") pod \"maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:18.912942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:18.912923 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr7r6\" (UniqueName: \"kubernetes.io/projected/7c6b648f-9f86-4064-9890-8b653d7539b8-kube-api-access-dr7r6\") pod \"maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:18.962428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:18.962390 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:19.014477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:19.014439 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-metrics-certs\") pod \"maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:19.014647 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:19.014492 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6b648f-9f86-4064-9890-8b653d7539b8-openshift-service-ca\") pod \"maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:19.014647 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:19.014583 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-maas-api-tls\") pod \"maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:19.014742 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:19.014645 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr7r6\" (UniqueName: \"kubernetes.io/projected/7c6b648f-9f86-4064-9890-8b653d7539b8-kube-api-access-dr7r6\") pod \"maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:19.015422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:19.015396 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6b648f-9f86-4064-9890-8b653d7539b8-openshift-service-ca\") pod \"maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:19.017764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:19.017734 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-maas-api-tls\") pod \"maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:19.017944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:19.017890 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-metrics-certs\") pod \"maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:19.115000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:19.113293 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:19.115363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:19.115286 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr7r6\" (UniqueName: \"kubernetes.io/projected/7c6b648f-9f86-4064-9890-8b653d7539b8-kube-api-access-dr7r6\") pod \"maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:19.169927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:19.165215 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:19.612220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:19.612176 4828 patch_prober.go:29] interesting pod/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 17:09:19.612634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:19.612239 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 17:09:19.697466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:19.696006 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2"] Oct 30 17:09:19.707052 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:09:19.707018 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6b648f_9f86_4064_9890_8b653d7539b8.slice/crio-806621f2ec9a5c5e1132f6f2d5614a00bb3070e06d1e6dabb698122978babeb8 WatchSource:0}: Error finding container 806621f2ec9a5c5e1132f6f2d5614a00bb3070e06d1e6dabb698122978babeb8: Status 404 returned error can't find the container with id 806621f2ec9a5c5e1132f6f2d5614a00bb3070e06d1e6dabb698122978babeb8 Oct 30 17:09:20.606922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:20.606744 4828 patch_prober.go:29] interesting pod/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 17:09:20.606922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:20.606803 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 17:09:20.672643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:20.672611 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:20.680638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:20.680585 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" event={"ID":"7c6b648f-9f86-4064-9890-8b653d7539b8","Type":"ContainerStarted","Data":"7a4e6a074469f8baeffd911d24cb6754bd22f2deaca54a61e42bcea44d3e79e6"} Oct 30 17:09:20.717108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:20.680975 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" event={"ID":"7c6b648f-9f86-4064-9890-8b653d7539b8","Type":"ContainerStarted","Data":"806621f2ec9a5c5e1132f6f2d5614a00bb3070e06d1e6dabb698122978babeb8"} Oct 30 17:09:20.727447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:20.727358 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" podStartSLOduration=2.323202085 podStartE2EDuration="2.727339751s" totalImagesPullingTime="404.137666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:20.719239695 +0000 UTC m=+1338.364749874" watchObservedRunningTime="2026-10-30 17:09:20.727339751 +0000 UTC m=+1338.372849932" Oct 30 17:09:21.435008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:21.434634 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:21.602922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:21.600024 4828 patch_prober.go:29] interesting pod/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 17:09:21.602922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:21.600092 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 17:09:22.591560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:22.591523 4828 patch_prober.go:29] interesting pod/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Oct 30 17:09:22.592132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:22.591592 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Oct 30 17:09:23.073525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:23.073482 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:23.596542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:23.596512 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:09:23.597064 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:23.597001 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:09:23.597761 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:23.597741 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:09:24.749665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:24.749627 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:26.608923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:26.595995 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:27.730151 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:27.730115 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:09:28.359521 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:28.357360 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:30.117922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:30.112046 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:31.715580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:31.715539 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:33.341040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:33.341002 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:35.311299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:35.311262 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:35.482299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:35.482263 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:37.277537 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:37.277485 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:37.913834 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:09:37.913793 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:09:38.915548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:38.915506 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:40.721983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:40.721938 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:42.207321 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:42.207285 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:42.391929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:42.377432 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:43.593483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:43.593450 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8659-a6d87b"] Oct 30 17:09:43.626230 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:43.626198 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8659-a6d87b" Oct 30 17:09:43.646003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:43.645968 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wddsm\"" Oct 30 17:09:43.686127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:43.677841 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8659-a6d87b"] Oct 30 17:09:43.730307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:43.706247 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdhfr\" (UniqueName: \"kubernetes.io/projected/2e2b61aa-eb33-4d89-b174-5f3c9b6104b8-kube-api-access-kdhfr\") pod \"test-curl-8659-a6d87b\" (UID: \"2e2b61aa-eb33-4d89-b174-5f3c9b6104b8\") " pod="opendatahub/test-curl-8659-a6d87b" Oct 30 17:09:43.832164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:43.832113 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdhfr\" (UniqueName: \"kubernetes.io/projected/2e2b61aa-eb33-4d89-b174-5f3c9b6104b8-kube-api-access-kdhfr\") pod \"test-curl-8659-a6d87b\" (UID: \"2e2b61aa-eb33-4d89-b174-5f3c9b6104b8\") " pod="opendatahub/test-curl-8659-a6d87b" Oct 30 17:09:43.857326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:43.857252 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdhfr\" (UniqueName: \"kubernetes.io/projected/2e2b61aa-eb33-4d89-b174-5f3c9b6104b8-kube-api-access-kdhfr\") pod \"test-curl-8659-a6d87b\" (UID: \"2e2b61aa-eb33-4d89-b174-5f3c9b6104b8\") " pod="opendatahub/test-curl-8659-a6d87b" Oct 30 17:09:44.106651 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:44.106611 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8659-a6d87b" Oct 30 17:09:44.158616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:44.158557 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:44.450164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:44.443259 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8659-a6d87b"] Oct 30 17:09:45.091682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:45.091645 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8659-a6d87b" event={"ID":"2e2b61aa-eb33-4d89-b174-5f3c9b6104b8","Type":"ContainerStarted","Data":"8adc2339dd3a695ab02b8a3c82e763c8fd7e361cb8172d210650085d788b70e7"} Oct 30 17:09:45.934714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:45.934675 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:47.519926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:47.510479 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:49.106918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:49.106860 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:50.938060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:50.938020 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:53.150061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:53.149974 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:53.269941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:53.248915 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8659-a6d87b" event={"ID":"2e2b61aa-eb33-4d89-b174-5f3c9b6104b8","Type":"ContainerStarted","Data":"c4ee9941f858adec2eee6b362bf42ded62324782d00b218416d3b233ce7b1f3e"} Oct 30 17:09:53.319087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:53.318975 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8659-a6d87b" podStartSLOduration=1.76484783 podStartE2EDuration="10.318955101s" totalImagesPullingTime="8.554107271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:53.312703352 +0000 UTC m=+1370.958213550" watchObservedRunningTime="2026-10-30 17:09:53.318955101 +0000 UTC m=+1370.964465284" Oct 30 17:09:53.666687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:53.666534 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:54.802560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:54.802517 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8659-a6d87b"] Oct 30 17:09:54.809182 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:54.809151 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8659-a6d87b"] Oct 30 17:09:55.068251 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:55.068213 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:55.440490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:55.440391 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8659-a6d87b" podUID="2e2b61aa-eb33-4d89-b174-5f3c9b6104b8" containerName="test-curl-8659-a6d87b" containerID="cri-o://c4ee9941f858adec2eee6b362bf42ded62324782d00b218416d3b233ce7b1f3e" gracePeriod=2 Oct 30 17:09:55.455203 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:09:55.455157 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8659-a6d87b\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="2e2b61aa-eb33-4d89-b174-5f3c9b6104b8" pod="opendatahub/test-curl-8659-a6d87b" Oct 30 17:09:55.942025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:55.941982 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:56.239770 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:56.239682 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:56.408147 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:56.408097 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:57.041925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:57.041826 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:57.198183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:57.198142 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:57.515850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:57.515743 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:57.942794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:57.942759 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:09:58.315320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:58.315288 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8659-a6d87b" Oct 30 17:09:58.339309 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:58.339276 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2b61aa-eb33-4d89-b174-5f3c9b6104b8-kube-api-access-kdhfr\" (UniqueName: \"kubernetes.io/projected/2e2b61aa-eb33-4d89-b174-5f3c9b6104b8-kube-api-access-kdhfr\") pod \"2e2b61aa-eb33-4d89-b174-5f3c9b6104b8\" (UID: \"2e2b61aa-eb33-4d89-b174-5f3c9b6104b8\") " Oct 30 17:09:58.341605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:58.341577 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2b61aa-eb33-4d89-b174-5f3c9b6104b8-kube-api-access-kdhfr" pod "2e2b61aa-eb33-4d89-b174-5f3c9b6104b8" (UID: "2e2b61aa-eb33-4d89-b174-5f3c9b6104b8"). InnerVolumeSpecName "kube-api-access-kdhfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:58.440984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:58.440942 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdhfr\" (UniqueName: \"kubernetes.io/projected/2e2b61aa-eb33-4d89-b174-5f3c9b6104b8-kube-api-access-kdhfr\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:09:58.485833 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:58.485793 4828 generic.go:356] "Generic (PLEG): container finished" podID="2e2b61aa-eb33-4d89-b174-5f3c9b6104b8" containerID="c4ee9941f858adec2eee6b362bf42ded62324782d00b218416d3b233ce7b1f3e" exitCode=137 Oct 30 17:09:58.486082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:58.485940 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8659-a6d87b" Oct 30 17:09:58.499660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:58.499623 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2b61aa-eb33-4d89-b174-5f3c9b6104b8" path="/var/lib/kubelet/pods/2e2b61aa-eb33-4d89-b174-5f3c9b6104b8/volumes" Oct 30 17:09:58.500357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:58.500340 4828 scope.go:122] "RemoveContainer" containerID="c4ee9941f858adec2eee6b362bf42ded62324782d00b218416d3b233ce7b1f3e" Oct 30 17:09:58.513389 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:09:58.513346 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8659-a6d87b\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="2e2b61aa-eb33-4d89-b174-5f3c9b6104b8" pod="opendatahub/test-curl-8659-a6d87b" Oct 30 17:09:58.514644 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:58.514621 4828 scope.go:122] "RemoveContainer" containerID="c4ee9941f858adec2eee6b362bf42ded62324782d00b218416d3b233ce7b1f3e" Oct 30 17:09:58.515105 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:09:58.515081 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ee9941f858adec2eee6b362bf42ded62324782d00b218416d3b233ce7b1f3e\": container with ID starting with c4ee9941f858adec2eee6b362bf42ded62324782d00b218416d3b233ce7b1f3e not found: ID does not exist" containerID="c4ee9941f858adec2eee6b362bf42ded62324782d00b218416d3b233ce7b1f3e" Oct 30 17:09:58.515201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:58.515118 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ee9941f858adec2eee6b362bf42ded62324782d00b218416d3b233ce7b1f3e"} err="failed to get container status \"c4ee9941f858adec2eee6b362bf42ded62324782d00b218416d3b233ce7b1f3e\": rpc error: code = NotFound desc = could not find container \"c4ee9941f858adec2eee6b362bf42ded62324782d00b218416d3b233ce7b1f3e\": container with ID starting with c4ee9941f858adec2eee6b362bf42ded62324782d00b218416d3b233ce7b1f3e not found: ID does not exist" Oct 30 17:09:58.907089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:09:58.907050 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:00.805942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:00.799846 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:02.122861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:02.122819 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:02.333790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:02.333755 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:02.754660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:02.752522 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:03.813060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:03.813019 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:03.987465 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:03.987424 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:04.425930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.421045 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn"] Oct 30 17:10:04.425930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.422471 4828 cpu_manager.go:414] "containerMap: removing container" podUID="2e2b61aa-eb33-4d89-b174-5f3c9b6104b8" containerName="test-curl-8659-a6d87b" Oct 30 17:10:04.425930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.422497 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2b61aa-eb33-4d89-b174-5f3c9b6104b8" containerName="test-curl-8659-a6d87b" Oct 30 17:10:04.425930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.422784 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2b61aa-eb33-4d89-b174-5f3c9b6104b8" containerName="test-curl-8659-a6d87b" Oct 30 17:10:04.529141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.529103 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" Oct 30 17:10:04.562165 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.562124 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfbd42e-a9a9-4d16-b165-aa64bb079056-plugins-config-volume\") pod \"payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn\" (UID: \"1cfbd42e-a9a9-4d16-b165-aa64bb079056\") " pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" Oct 30 17:10:04.562696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.562670 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq2s4\" (UniqueName: \"kubernetes.io/projected/1cfbd42e-a9a9-4d16-b165-aa64bb079056-kube-api-access-hq2s4\") pod \"payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn\" (UID: \"1cfbd42e-a9a9-4d16-b165-aa64bb079056\") " pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" Oct 30 17:10:04.591751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.591715 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-362bbf\"" Oct 30 17:10:04.593523 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.592559 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-362bbf-dockercfg-bvqdd\"" Oct 30 17:10:04.614652 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.614624 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn"] Oct 30 17:10:04.665518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.665475 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq2s4\" (UniqueName: \"kubernetes.io/projected/1cfbd42e-a9a9-4d16-b165-aa64bb079056-kube-api-access-hq2s4\") pod \"payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn\" (UID: \"1cfbd42e-a9a9-4d16-b165-aa64bb079056\") " pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" Oct 30 17:10:04.665702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.665672 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfbd42e-a9a9-4d16-b165-aa64bb079056-plugins-config-volume\") pod \"payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn\" (UID: \"1cfbd42e-a9a9-4d16-b165-aa64bb079056\") " pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" Oct 30 17:10:04.666585 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.666561 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfbd42e-a9a9-4d16-b165-aa64bb079056-plugins-config-volume\") pod \"payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn\" (UID: \"1cfbd42e-a9a9-4d16-b165-aa64bb079056\") " pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" Oct 30 17:10:04.696618 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.696527 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq2s4\" (UniqueName: \"kubernetes.io/projected/1cfbd42e-a9a9-4d16-b165-aa64bb079056-kube-api-access-hq2s4\") pod \"payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn\" (UID: \"1cfbd42e-a9a9-4d16-b165-aa64bb079056\") " pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" Oct 30 17:10:04.897937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.897882 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:04.955053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:04.937210 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" Oct 30 17:10:05.331504 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:10:05.331467 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfbd42e_a9a9_4d16_b165_aa64bb079056.slice/crio-3c3aa50d90b88a0c70891930e59f691d5dacfd96a50e19a5fe14583659d4ce79 WatchSource:0}: Error finding container 3c3aa50d90b88a0c70891930e59f691d5dacfd96a50e19a5fe14583659d4ce79: Status 404 returned error can't find the container with id 3c3aa50d90b88a0c70891930e59f691d5dacfd96a50e19a5fe14583659d4ce79 Oct 30 17:10:05.332753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:05.332718 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn"] Oct 30 17:10:05.525695 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:05.525613 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw"] Oct 30 17:10:05.560440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:05.560401 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6"] Oct 30 17:10:05.632099 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:05.632016 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" event={"ID":"1cfbd42e-a9a9-4d16-b165-aa64bb079056","Type":"ContainerStarted","Data":"a4e82e0555c02e98442e9d407c1ccbabeb1ace017853e00a076366e7e029df96"} Oct 30 17:10:05.633534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:05.633502 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" event={"ID":"1cfbd42e-a9a9-4d16-b165-aa64bb079056","Type":"ContainerStarted","Data":"3c3aa50d90b88a0c70891930e59f691d5dacfd96a50e19a5fe14583659d4ce79"} Oct 30 17:10:05.633705 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:05.633547 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" Oct 30 17:10:05.677394 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:05.677335 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" podStartSLOduration=1.677318508 podStartE2EDuration="1.677318508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:05.675921816 +0000 UTC m=+1383.321431995" watchObservedRunningTime="2026-10-30 17:10:05.677318508 +0000 UTC m=+1383.322828690" Oct 30 17:10:05.706128 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:05.706091 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:05.876112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:05.876051 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:07.279941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:07.277254 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:07.916628 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:10:07.916574 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:10:09.109930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:09.106635 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:10.526665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:10.526621 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" podUID="cdb4b388-a68c-454b-92fc-34a9c1b94a9f" containerName="payload-processing" containerID="cri-o://94e641bcbde48266b4a025e26f0e0b786caa742d962235adef8390b2c1b2c2a6" gracePeriod=25 Oct 30 17:10:10.561455 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:10.561390 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" podUID="4740747b-0ca9-4643-910f-32507d5d0bed" containerName="payload-pre-processing" containerID="cri-o://3c26742939ff7b68081c51a7f4e22c4705382f4281cff7152e1df85645298fac" gracePeriod=25 Oct 30 17:10:10.896111 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:10.896014 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:12.495978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:12.495934 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:12.763993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:12.761591 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:13.202675 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.202573 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:13.774014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.773969 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8"] Oct 30 17:10:13.818267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.817024 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:13.834077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.834012 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c4c2e4-openshift-default-dockercfg-7cm86\"" Oct 30 17:10:13.834567 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.834543 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:13.901367 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.901328 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068993d6-1de9-4224-b028-39ce4fb563ea-istiod-ca-cert\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:13.901646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.901624 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-envoy\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:13.901780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.901766 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-istio-token\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:13.901890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.901877 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-data\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:13.903069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.902176 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-certs\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:13.903069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.902275 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-credential-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:13.903069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.902541 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4jc6\" (UniqueName: \"kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-kube-api-access-f4jc6\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:13.903069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.902629 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:13.903069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.902712 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068993d6-1de9-4224-b028-39ce4fb563ea-istio-podinfo\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:13.958923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:13.949167 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8"] Oct 30 17:10:14.014956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.004430 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068993d6-1de9-4224-b028-39ce4fb563ea-istio-podinfo\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.017831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.015325 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068993d6-1de9-4224-b028-39ce4fb563ea-istiod-ca-cert\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.018174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.018142 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-envoy\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.018335 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.018318 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-istio-token\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.018462 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.018448 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-data\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.018612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.018598 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-certs\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.018775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.018760 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-credential-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.019217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.019192 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4jc6\" (UniqueName: \"kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-kube-api-access-f4jc6\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.019509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.019487 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.020351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.020326 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.021243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.021221 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068993d6-1de9-4224-b028-39ce4fb563ea-istiod-ca-cert\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.022653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.022628 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-data\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.023318 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.023050 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-credential-socket\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.023444 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.023267 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-certs\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.045983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.044292 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:14.048779 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.048716 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-envoy\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.209977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.160692 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4jc6\" (UniqueName: \"kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-kube-api-access-f4jc6\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.209977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.175758 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" podUID="cdb4b388-a68c-454b-92fc-34a9c1b94a9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:10:14.209977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.209183 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-istio-token\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.226986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.220582 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068993d6-1de9-4224-b028-39ce4fb563ea-istio-podinfo\") pod \"e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.425395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.400516 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" podUID="4740747b-0ca9-4643-910f-32507d5d0bed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:10:14.492782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.492744 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:14.846891 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:10:14.846844 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068993d6_1de9_4224_b028_39ce4fb563ea.slice/crio-c51dff39fb31b1165a9d700c2517277838f28f8e6f1b6976b907a6db0aec3c00 WatchSource:0}: Error finding container c51dff39fb31b1165a9d700c2517277838f28f8e6f1b6976b907a6db0aec3c00: Status 404 returned error can't find the container with id c51dff39fb31b1165a9d700c2517277838f28f8e6f1b6976b907a6db0aec3c00 Oct 30 17:10:14.848504 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.848478 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8"] Oct 30 17:10:14.850601 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.850565 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:10:14.850712 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.850634 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:10:14.850712 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.850664 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:10:14.964925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:14.955134 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:15.011566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:15.011525 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" event={"ID":"068993d6-1de9-4224-b028-39ce4fb563ea","Type":"ContainerStarted","Data":"c51dff39fb31b1165a9d700c2517277838f28f8e6f1b6976b907a6db0aec3c00"} Oct 30 17:10:16.023582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:16.023545 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" event={"ID":"068993d6-1de9-4224-b028-39ce4fb563ea","Type":"ContainerStarted","Data":"aef9d048db08610705874f6def9ab3b811be2246deb4ac2d7995304da7b878a9"} Oct 30 17:10:16.086036 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:16.085957 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" podStartSLOduration=3.085939629 podStartE2EDuration="3.085939629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:16.082411078 +0000 UTC m=+1393.727921259" watchObservedRunningTime="2026-10-30 17:10:16.085939629 +0000 UTC m=+1393.731449808" Oct 30 17:10:16.587305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:16.586652 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:16.587305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:16.586781 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:16.826491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:16.826460 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" Oct 30 17:10:17.033184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:17.033154 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:17.034428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:17.034405 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:10:17.161770 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:17.161736 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:18.638943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:18.638851 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:18.793293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:18.793241 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:19.041080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:19.041045 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:19.975000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:19.974952 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:20.285486 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:20.285448 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:20.451208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:20.451136 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:20.632981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:20.623646 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:21.021016 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:21.020977 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:21.267537 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:21.267505 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb"] Oct 30 17:10:21.271308 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:21.271226 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:21.275104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:21.275014 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5"] Oct 30 17:10:21.426553 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:21.426510 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:21.618999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:21.592015 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:23.379631 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:23.379591 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:24.116146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:24.116106 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" podUID="cdb4b388-a68c-454b-92fc-34a9c1b94a9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:10:24.116358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:24.116112 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" podUID="4740747b-0ca9-4643-910f-32507d5d0bed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:10:24.440810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:24.440725 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh"] Oct 30 17:10:24.441273 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:24.440988 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" podUID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" containerName="istio-proxy" containerID="cri-o://a47fbe97f36fd01e63534a593c45ee452441fd2dec4051b9ccad7bbc01dbc848" gracePeriod=30 Oct 30 17:10:24.454022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:24.453983 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt"] Oct 30 17:10:24.454303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:24.454249 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerName="istio-proxy" containerID="cri-o://ec827807ba76bfab23adc5a6fc8ba8c146b5e59467bbd341f963a894e3a24c23" gracePeriod=30 Oct 30 17:10:24.461603 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:24.461571 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw"] Oct 30 17:10:24.461839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:24.461803 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" podUID="1fa37ccf-2a69-4083-a6db-b32cd13132a4" containerName="istio-proxy" containerID="cri-o://1c3f9ef40f2bf401e648b20c0b5b472c1735b410e843f94ad0b1104f3d394129" gracePeriod=30 Oct 30 17:10:24.563762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:24.563727 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:24.718094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:24.718005 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:25.591292 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:25.591250 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:26.268626 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:26.268578 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" podUID="77d5e34f-129b-48a3-9aa3-e8439e81b6db" containerName="payload-processing" containerID="cri-o://a2afb8eeb4798e831dacc3003ed8bdad84c49f0698c3117c2f711bd2b7668f6d" gracePeriod=25 Oct 30 17:10:26.276160 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:26.276123 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" podUID="d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d" containerName="payload-pre-processing" containerID="cri-o://564fec47d6be2160f4d977e2675b149ed465e6caba81f3e49d71f953dc01645f" gracePeriod=25 Oct 30 17:10:26.392578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:26.392543 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:26.565365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:26.565230 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:26.846719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:26.846637 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:27.028951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.027141 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" Oct 30 17:10:27.028951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.028656 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d5e34f-129b-48a3-9aa3-e8439e81b6db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d5e34f-129b-48a3-9aa3-e8439e81b6db-plugins-config-volume\") pod \"77d5e34f-129b-48a3-9aa3-e8439e81b6db\" (UID: \"77d5e34f-129b-48a3-9aa3-e8439e81b6db\") " Oct 30 17:10:27.034160 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.034121 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d5e34f-129b-48a3-9aa3-e8439e81b6db-plugins-config-volume" pod "77d5e34f-129b-48a3-9aa3-e8439e81b6db" (UID: "77d5e34f-129b-48a3-9aa3-e8439e81b6db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:27.036084 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.028895 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d5e34f-129b-48a3-9aa3-e8439e81b6db-kube-api-access-hfk7k\" (UniqueName: \"kubernetes.io/projected/77d5e34f-129b-48a3-9aa3-e8439e81b6db-kube-api-access-hfk7k\") pod \"77d5e34f-129b-48a3-9aa3-e8439e81b6db\" (UID: \"77d5e34f-129b-48a3-9aa3-e8439e81b6db\") " Oct 30 17:10:27.042706 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.042655 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77d5e34f-129b-48a3-9aa3-e8439e81b6db-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:27.071518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.052705 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d5e34f-129b-48a3-9aa3-e8439e81b6db-kube-api-access-hfk7k" pod "77d5e34f-129b-48a3-9aa3-e8439e81b6db" (UID: "77d5e34f-129b-48a3-9aa3-e8439e81b6db"). InnerVolumeSpecName "kube-api-access-hfk7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:27.071518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.062236 4828 generic.go:356] "Generic (PLEG): container finished" podID="d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d" containerID="564fec47d6be2160f4d977e2675b149ed465e6caba81f3e49d71f953dc01645f" exitCode=0 Oct 30 17:10:27.082989 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.075008 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" event={"ID":"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d","Type":"ContainerDied","Data":"564fec47d6be2160f4d977e2675b149ed465e6caba81f3e49d71f953dc01645f"} Oct 30 17:10:27.082989 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.080957 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" Oct 30 17:10:27.098103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.094287 4828 generic.go:356] "Generic (PLEG): container finished" podID="77d5e34f-129b-48a3-9aa3-e8439e81b6db" containerID="a2afb8eeb4798e831dacc3003ed8bdad84c49f0698c3117c2f711bd2b7668f6d" exitCode=0 Oct 30 17:10:27.098103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.094386 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" event={"ID":"77d5e34f-129b-48a3-9aa3-e8439e81b6db","Type":"ContainerDied","Data":"a2afb8eeb4798e831dacc3003ed8bdad84c49f0698c3117c2f711bd2b7668f6d"} Oct 30 17:10:27.098103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.094445 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" event={"ID":"77d5e34f-129b-48a3-9aa3-e8439e81b6db","Type":"ContainerDied","Data":"8be46d09136005473260051e99dc6b7fe4868e71b5d1367e76e6a47d9356403e"} Oct 30 17:10:27.098103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.094477 4828 scope.go:122] "RemoveContainer" containerID="a2afb8eeb4798e831dacc3003ed8bdad84c49f0698c3117c2f711bd2b7668f6d" Oct 30 17:10:27.098103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.094666 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" Oct 30 17:10:27.194176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.194127 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-kube-api-access-cmplc\" (UniqueName: \"kubernetes.io/projected/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-kube-api-access-cmplc\") pod \"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d\" (UID: \"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d\") " Oct 30 17:10:27.195406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.195372 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-plugins-config-volume\") pod \"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d\" (UID: \"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d\") " Oct 30 17:10:27.200562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.200527 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfk7k\" (UniqueName: \"kubernetes.io/projected/77d5e34f-129b-48a3-9aa3-e8439e81b6db-kube-api-access-hfk7k\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:27.215767 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.215698 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-plugins-config-volume" pod "d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d" (UID: "d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:27.307778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.307694 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:27.337088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.337066 4828 scope.go:122] "RemoveContainer" containerID="a2afb8eeb4798e831dacc3003ed8bdad84c49f0698c3117c2f711bd2b7668f6d" Oct 30 17:10:27.337567 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.337545 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-kube-api-access-cmplc" pod "d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d" (UID: "d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d"). InnerVolumeSpecName "kube-api-access-cmplc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:27.338325 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:10:27.338304 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2afb8eeb4798e831dacc3003ed8bdad84c49f0698c3117c2f711bd2b7668f6d\": container with ID starting with a2afb8eeb4798e831dacc3003ed8bdad84c49f0698c3117c2f711bd2b7668f6d not found: ID does not exist" containerID="a2afb8eeb4798e831dacc3003ed8bdad84c49f0698c3117c2f711bd2b7668f6d" Oct 30 17:10:27.338627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.338599 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2afb8eeb4798e831dacc3003ed8bdad84c49f0698c3117c2f711bd2b7668f6d"} err="failed to get container status \"a2afb8eeb4798e831dacc3003ed8bdad84c49f0698c3117c2f711bd2b7668f6d\": rpc error: code = NotFound desc = could not find container \"a2afb8eeb4798e831dacc3003ed8bdad84c49f0698c3117c2f711bd2b7668f6d\": container with ID starting with a2afb8eeb4798e831dacc3003ed8bdad84c49f0698c3117c2f711bd2b7668f6d not found: ID does not exist" Oct 30 17:10:27.397256 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.397160 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb"] Oct 30 17:10:27.406009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.405965 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb"] Oct 30 17:10:27.409275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.409242 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmplc\" (UniqueName: \"kubernetes.io/projected/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d-kube-api-access-cmplc\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:27.543130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.543083 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" podUID="d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.119:50052\" within 1s: context deadline exceeded" Oct 30 17:10:27.543130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:27.543115 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-c425a3e3-754d764588-ssgwb" podUID="77d5e34f-129b-48a3-9aa3-e8439e81b6db" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.121:50052\" within 1s: context deadline exceeded" Oct 30 17:10:27.543336 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:10:27.543176 4828 logging.go:55] [core] [Channel #2054 SubChannel #2057]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.119:50052", ServerName: "10.132.0.119:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.119:50052: operation was canceled" Oct 30 17:10:27.543336 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:10:27.543242 4828 logging.go:55] [core] [Channel #2055 SubChannel #2056]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.121:50052", ServerName: "10.132.0.121:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.121:50052: operation was canceled" Oct 30 17:10:28.115207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.111236 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" event={"ID":"d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d","Type":"ContainerDied","Data":"68afacca90c3161d47290ef192870d799d6e6eed37fa95b039343e86bda502f9"} Oct 30 17:10:28.115207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.111250 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5" Oct 30 17:10:28.115207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.111325 4828 scope.go:122] "RemoveContainer" containerID="564fec47d6be2160f4d977e2675b149ed465e6caba81f3e49d71f953dc01645f" Oct 30 17:10:28.174220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.174174 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5"] Oct 30 17:10:28.185686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.185647 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:28.192500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.188393 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c425a3e3-9fb58b895-wnxp5"] Oct 30 17:10:28.343396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.343356 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:28.368549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.368471 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8"] Oct 30 17:10:28.369433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.369408 4828 cpu_manager.go:414] "containerMap: removing container" podUID="77d5e34f-129b-48a3-9aa3-e8439e81b6db" containerName="payload-processing" Oct 30 17:10:28.369433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.369429 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d5e34f-129b-48a3-9aa3-e8439e81b6db" containerName="payload-processing" Oct 30 17:10:28.369637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.369482 4828 cpu_manager.go:414] "containerMap: removing container" podUID="d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d" containerName="payload-pre-processing" Oct 30 17:10:28.369637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.369488 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d" containerName="payload-pre-processing" Oct 30 17:10:28.369637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.369614 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d" containerName="payload-pre-processing" Oct 30 17:10:28.369637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.369630 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d5e34f-129b-48a3-9aa3-e8439e81b6db" containerName="payload-processing" Oct 30 17:10:28.376053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.376031 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" Oct 30 17:10:28.392488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.392459 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-39fecf\"" Oct 30 17:10:28.393250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.392661 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-39fecf-dockercfg-2kvww\"" Oct 30 17:10:28.394941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.394915 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8"] Oct 30 17:10:28.744184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.737454 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxbcf\" (UniqueName: \"kubernetes.io/projected/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-kube-api-access-lxbcf\") pod \"payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8\" (UID: \"c83f9c48-b4d3-49fb-abc9-36c466cc00f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" Oct 30 17:10:28.744184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.737745 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8\" (UID: \"c83f9c48-b4d3-49fb-abc9-36c466cc00f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" Oct 30 17:10:28.747199 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.746454 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d5e34f-129b-48a3-9aa3-e8439e81b6db" path="/var/lib/kubelet/pods/77d5e34f-129b-48a3-9aa3-e8439e81b6db/volumes" Oct 30 17:10:28.757976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.747319 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d" path="/var/lib/kubelet/pods/d1a5cf9c-0f32-4a03-9d5c-2a2764bbc47d/volumes" Oct 30 17:10:28.757976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.747890 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk"] Oct 30 17:10:28.760476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.760450 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" Oct 30 17:10:28.998993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:28.978964 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk"] Oct 30 17:10:29.002912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.002855 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8\" (UID: \"c83f9c48-b4d3-49fb-abc9-36c466cc00f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" Oct 30 17:10:29.004614 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.004585 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxbcf\" (UniqueName: \"kubernetes.io/projected/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-kube-api-access-lxbcf\") pod \"payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8\" (UID: \"c83f9c48-b4d3-49fb-abc9-36c466cc00f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" Oct 30 17:10:29.005624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.005603 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99192812-9c04-4cfc-a866-f4f0afee30de-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk\" (UID: \"99192812-9c04-4cfc-a866-f4f0afee30de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" Oct 30 17:10:29.006221 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.006199 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqk6t\" (UniqueName: \"kubernetes.io/projected/99192812-9c04-4cfc-a866-f4f0afee30de-kube-api-access-rqk6t\") pod \"payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk\" (UID: \"99192812-9c04-4cfc-a866-f4f0afee30de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" Oct 30 17:10:29.006593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.003925 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8\" (UID: \"c83f9c48-b4d3-49fb-abc9-36c466cc00f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" Oct 30 17:10:29.029596 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.029565 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxbcf\" (UniqueName: \"kubernetes.io/projected/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-kube-api-access-lxbcf\") pod \"payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8\" (UID: \"c83f9c48-b4d3-49fb-abc9-36c466cc00f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" Oct 30 17:10:29.107684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.107643 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99192812-9c04-4cfc-a866-f4f0afee30de-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk\" (UID: \"99192812-9c04-4cfc-a866-f4f0afee30de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" Oct 30 17:10:29.107684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.107691 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqk6t\" (UniqueName: \"kubernetes.io/projected/99192812-9c04-4cfc-a866-f4f0afee30de-kube-api-access-rqk6t\") pod \"payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk\" (UID: \"99192812-9c04-4cfc-a866-f4f0afee30de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" Oct 30 17:10:29.108430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.108403 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99192812-9c04-4cfc-a866-f4f0afee30de-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk\" (UID: \"99192812-9c04-4cfc-a866-f4f0afee30de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" Oct 30 17:10:29.132991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.132956 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqk6t\" (UniqueName: \"kubernetes.io/projected/99192812-9c04-4cfc-a866-f4f0afee30de-kube-api-access-rqk6t\") pod \"payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk\" (UID: \"99192812-9c04-4cfc-a866-f4f0afee30de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" Oct 30 17:10:29.297098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.297051 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" Oct 30 17:10:29.307938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.306762 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" Oct 30 17:10:29.657803 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:10:29.657765 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83f9c48_b4d3_49fb_abc9_36c466cc00f2.slice/crio-450a55b9d147a8d24c525ace24702cc3497984c6dc8fd83e04106b0af088992e WatchSource:0}: Error finding container 450a55b9d147a8d24c525ace24702cc3497984c6dc8fd83e04106b0af088992e: Status 404 returned error can't find the container with id 450a55b9d147a8d24c525ace24702cc3497984c6dc8fd83e04106b0af088992e Oct 30 17:10:29.661511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.661484 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8"] Oct 30 17:10:29.715396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.715358 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk"] Oct 30 17:10:29.830611 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.830269 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:10:29.860842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.860818 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:10:29.929209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.928844 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-envoy\") pod \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " Oct 30 17:10:29.929209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.928921 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-data\") pod \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " Oct 30 17:10:29.929209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.928959 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-credential-socket\") pod \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " Oct 30 17:10:29.929209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.928989 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-podinfo\") pod \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " Oct 30 17:10:29.929209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929026 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-envoy\") pod \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " Oct 30 17:10:29.929209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929073 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istiod-ca-cert\") pod \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " Oct 30 17:10:29.929209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929152 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istiod-ca-cert\") pod \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " Oct 30 17:10:29.929573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929242 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-certs\") pod \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " Oct 30 17:10:29.929573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929255 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-data" pod "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" (UID: "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:29.929573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929286 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-podinfo\") pod \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " Oct 30 17:10:29.929573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929371 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-token\" (UniqueName: \"kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-token\") pod \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " Oct 30 17:10:29.929573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929457 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-socket\") pod \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " Oct 30 17:10:29.929573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929533 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-certs\") pod \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " Oct 30 17:10:29.929573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929560 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-data\") pod \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " Oct 30 17:10:29.929875 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929686 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-credential-socket\") pod \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " Oct 30 17:10:29.929875 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929735 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-socket\") pod \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " Oct 30 17:10:29.929875 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929779 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-kube-api-access-p7rv5\" (UniqueName: \"kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-kube-api-access-p7rv5\") pod \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\" (UID: \"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d\") " Oct 30 17:10:29.929875 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929825 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-token\" (UniqueName: \"kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-token\") pod \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " Oct 30 17:10:29.930092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.929979 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-kube-api-access-69zc5\" (UniqueName: \"kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-kube-api-access-69zc5\") pod \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\" (UID: \"4546e53a-e9f2-4e09-ad63-c9df1bb57570\") " Oct 30 17:10:29.930156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.930136 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-data" pod "4546e53a-e9f2-4e09-ad63-c9df1bb57570" (UID: "4546e53a-e9f2-4e09-ad63-c9df1bb57570"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:29.930797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.930320 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-socket" pod "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" (UID: "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:29.930797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.930570 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-certs" pod "4546e53a-e9f2-4e09-ad63-c9df1bb57570" (UID: "4546e53a-e9f2-4e09-ad63-c9df1bb57570"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:29.930996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.930803 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-certs" pod "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" (UID: "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:29.931360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.931227 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:29.931360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.931257 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:29.931360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.931271 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:29.931360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.931283 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:29.931360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.931295 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:29.931676 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.931627 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istiod-ca-cert" pod "4546e53a-e9f2-4e09-ad63-c9df1bb57570" (UID: "4546e53a-e9f2-4e09-ad63-c9df1bb57570"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:29.932545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.931867 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-credential-socket" pod "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" (UID: "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:29.932545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.932084 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-socket" pod "4546e53a-e9f2-4e09-ad63-c9df1bb57570" (UID: "4546e53a-e9f2-4e09-ad63-c9df1bb57570"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:29.932545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.932261 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-credential-socket" pod "4546e53a-e9f2-4e09-ad63-c9df1bb57570" (UID: "4546e53a-e9f2-4e09-ad63-c9df1bb57570"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:29.935971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.932918 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istiod-ca-cert" pod "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" (UID: "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:29.935971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.933255 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-kube-api-access-69zc5" pod "4546e53a-e9f2-4e09-ad63-c9df1bb57570" (UID: "4546e53a-e9f2-4e09-ad63-c9df1bb57570"). InnerVolumeSpecName "kube-api-access-69zc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:29.935971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.934222 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-podinfo" pod "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" (UID: "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:10:29.935971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.935226 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-envoy" pod "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" (UID: "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:29.935971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.935231 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-kube-api-access-p7rv5" pod "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" (UID: "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d"). InnerVolumeSpecName "kube-api-access-p7rv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:29.935971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.935331 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-envoy" pod "4546e53a-e9f2-4e09-ad63-c9df1bb57570" (UID: "4546e53a-e9f2-4e09-ad63-c9df1bb57570"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:29.935971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.935864 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-podinfo" pod "4546e53a-e9f2-4e09-ad63-c9df1bb57570" (UID: "4546e53a-e9f2-4e09-ad63-c9df1bb57570"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:10:29.936430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.936094 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-token" pod "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" (UID: "6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:29.937007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.936977 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-token" pod "4546e53a-e9f2-4e09-ad63-c9df1bb57570" (UID: "4546e53a-e9f2-4e09-ad63-c9df1bb57570"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:29.942871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.942846 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:29.960939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:29.960861 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:10:30.032847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.032809 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-certs\") pod \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " Oct 30 17:10:30.033070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.032855 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-credential-socket\") pod \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " Oct 30 17:10:30.033144 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.033118 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-data\") pod \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " Oct 30 17:10:30.033285 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.033261 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istiod-ca-cert\") pod \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " Oct 30 17:10:30.033413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.033296 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-podinfo\") pod \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " Oct 30 17:10:30.033413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.033346 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-kube-api-access-9kjlp\" (UniqueName: \"kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-kube-api-access-9kjlp\") pod \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " Oct 30 17:10:30.033413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.033396 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-token\" (UniqueName: \"kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-token\") pod \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " Oct 30 17:10:30.033605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.033460 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-socket\") pod \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " Oct 30 17:10:30.033605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.033497 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-envoy\") pod \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\" (UID: \"1fa37ccf-2a69-4083-a6db-b32cd13132a4\") " Oct 30 17:10:30.034152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.033763 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-certs" pod "1fa37ccf-2a69-4083-a6db-b32cd13132a4" (UID: "1fa37ccf-2a69-4083-a6db-b32cd13132a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:30.034152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.033796 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-credential-socket" pod "1fa37ccf-2a69-4083-a6db-b32cd13132a4" (UID: "1fa37ccf-2a69-4083-a6db-b32cd13132a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:30.034152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.033831 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-data" pod "1fa37ccf-2a69-4083-a6db-b32cd13132a4" (UID: "1fa37ccf-2a69-4083-a6db-b32cd13132a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:30.034797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.034768 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-socket" pod "1fa37ccf-2a69-4083-a6db-b32cd13132a4" (UID: "1fa37ccf-2a69-4083-a6db-b32cd13132a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:30.035471 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035434 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035471 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035461 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035478 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7rv5\" (UniqueName: \"kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-kube-api-access-p7rv5\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035493 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035506 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69zc5\" (UniqueName: \"kubernetes.io/projected/4546e53a-e9f2-4e09-ad63-c9df1bb57570-kube-api-access-69zc5\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035520 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035533 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035545 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035557 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035569 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035582 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035594 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4546e53a-e9f2-4e09-ad63-c9df1bb57570-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035606 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035619 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035632 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.035648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035644 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.036571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.035977 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istiod-ca-cert" pod "1fa37ccf-2a69-4083-a6db-b32cd13132a4" (UID: "1fa37ccf-2a69-4083-a6db-b32cd13132a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:30.036839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.036816 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-kube-api-access-9kjlp" pod "1fa37ccf-2a69-4083-a6db-b32cd13132a4" (UID: "1fa37ccf-2a69-4083-a6db-b32cd13132a4"). InnerVolumeSpecName "kube-api-access-9kjlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:30.038248 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.038223 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-token" pod "1fa37ccf-2a69-4083-a6db-b32cd13132a4" (UID: "1fa37ccf-2a69-4083-a6db-b32cd13132a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:30.038886 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.038861 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-podinfo" pod "1fa37ccf-2a69-4083-a6db-b32cd13132a4" (UID: "1fa37ccf-2a69-4083-a6db-b32cd13132a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:10:30.039311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.039288 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-envoy" pod "1fa37ccf-2a69-4083-a6db-b32cd13132a4" (UID: "1fa37ccf-2a69-4083-a6db-b32cd13132a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:30.133880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.133848 4828 generic.go:356] "Generic (PLEG): container finished" podID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" containerID="a47fbe97f36fd01e63534a593c45ee452441fd2dec4051b9ccad7bbc01dbc848" exitCode=0 Oct 30 17:10:30.134362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.133943 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" Oct 30 17:10:30.134362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.133939 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" event={"ID":"4546e53a-e9f2-4e09-ad63-c9df1bb57570","Type":"ContainerDied","Data":"a47fbe97f36fd01e63534a593c45ee452441fd2dec4051b9ccad7bbc01dbc848"} Oct 30 17:10:30.134362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.134121 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh" event={"ID":"4546e53a-e9f2-4e09-ad63-c9df1bb57570","Type":"ContainerDied","Data":"cc0e7b3bc86a413a9c4e39bd3905499d10dddb23f53fafb1b1139096108a69b0"} Oct 30 17:10:30.134362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.134174 4828 scope.go:122] "RemoveContainer" containerID="a47fbe97f36fd01e63534a593c45ee452441fd2dec4051b9ccad7bbc01dbc848" Oct 30 17:10:30.135863 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.135838 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" event={"ID":"99192812-9c04-4cfc-a866-f4f0afee30de","Type":"ContainerStarted","Data":"33863d549bad9df8be49bd335f23e9b1a7d2a12524bc00da84471d5aff662306"} Oct 30 17:10:30.136062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.135888 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" event={"ID":"99192812-9c04-4cfc-a866-f4f0afee30de","Type":"ContainerStarted","Data":"d2f0a02d4def22361ee95836d68e3fbb06823e2c6a55d7c60890fba2a4d22a9b"} Oct 30 17:10:30.136062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.135962 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" Oct 30 17:10:30.137004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.136888 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.137004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.136924 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.137004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.136932 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.137004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.136940 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.137004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.136948 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fa37ccf-2a69-4083-a6db-b32cd13132a4-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.137004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.136956 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kjlp\" (UniqueName: \"kubernetes.io/projected/1fa37ccf-2a69-4083-a6db-b32cd13132a4-kube-api-access-9kjlp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.137743 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.137722 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" event={"ID":"c83f9c48-b4d3-49fb-abc9-36c466cc00f2","Type":"ContainerStarted","Data":"69142c7894feead9cbc4790727c6490886eaaa6edaf18fdf80fa5225779795cb"} Oct 30 17:10:30.137852 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.137757 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" event={"ID":"c83f9c48-b4d3-49fb-abc9-36c466cc00f2","Type":"ContainerStarted","Data":"450a55b9d147a8d24c525ace24702cc3497984c6dc8fd83e04106b0af088992e"} Oct 30 17:10:30.137852 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.137782 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" Oct 30 17:10:30.139260 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.139241 4828 generic.go:356] "Generic (PLEG): container finished" podID="1fa37ccf-2a69-4083-a6db-b32cd13132a4" containerID="1c3f9ef40f2bf401e648b20c0b5b472c1735b410e843f94ad0b1104f3d394129" exitCode=0 Oct 30 17:10:30.139376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.139293 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" Oct 30 17:10:30.139376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.139319 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" event={"ID":"1fa37ccf-2a69-4083-a6db-b32cd13132a4","Type":"ContainerDied","Data":"1c3f9ef40f2bf401e648b20c0b5b472c1735b410e843f94ad0b1104f3d394129"} Oct 30 17:10:30.139376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.139360 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw" event={"ID":"1fa37ccf-2a69-4083-a6db-b32cd13132a4","Type":"ContainerDied","Data":"065aa482507d73100954a773615bf59beb6a467a382871d738f6af2f35941ca4"} Oct 30 17:10:30.140945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.140922 4828 generic.go:356] "Generic (PLEG): container finished" podID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerID="ec827807ba76bfab23adc5a6fc8ba8c146b5e59467bbd341f963a894e3a24c23" exitCode=0 Oct 30 17:10:30.141043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.140978 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" Oct 30 17:10:30.141091 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.140981 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" event={"ID":"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d","Type":"ContainerDied","Data":"ec827807ba76bfab23adc5a6fc8ba8c146b5e59467bbd341f963a894e3a24c23"} Oct 30 17:10:30.141142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.141124 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt" event={"ID":"6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d","Type":"ContainerDied","Data":"e5705ae3763a8a670498eee6d6f28b0fc50b4b22c4d3c2c8dd335a7890b45d35"} Oct 30 17:10:30.147560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.147539 4828 scope.go:122] "RemoveContainer" containerID="a47fbe97f36fd01e63534a593c45ee452441fd2dec4051b9ccad7bbc01dbc848" Oct 30 17:10:30.148176 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:10:30.148152 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a47fbe97f36fd01e63534a593c45ee452441fd2dec4051b9ccad7bbc01dbc848\": container with ID starting with a47fbe97f36fd01e63534a593c45ee452441fd2dec4051b9ccad7bbc01dbc848 not found: ID does not exist" containerID="a47fbe97f36fd01e63534a593c45ee452441fd2dec4051b9ccad7bbc01dbc848" Oct 30 17:10:30.148255 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.148183 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47fbe97f36fd01e63534a593c45ee452441fd2dec4051b9ccad7bbc01dbc848"} err="failed to get container status \"a47fbe97f36fd01e63534a593c45ee452441fd2dec4051b9ccad7bbc01dbc848\": rpc error: code = NotFound desc = could not find container \"a47fbe97f36fd01e63534a593c45ee452441fd2dec4051b9ccad7bbc01dbc848\": container with ID starting with a47fbe97f36fd01e63534a593c45ee452441fd2dec4051b9ccad7bbc01dbc848 not found: ID does not exist" Oct 30 17:10:30.148255 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.148198 4828 scope.go:122] "RemoveContainer" containerID="1c3f9ef40f2bf401e648b20c0b5b472c1735b410e843f94ad0b1104f3d394129" Oct 30 17:10:30.159862 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.159838 4828 scope.go:122] "RemoveContainer" containerID="1c3f9ef40f2bf401e648b20c0b5b472c1735b410e843f94ad0b1104f3d394129" Oct 30 17:10:30.160292 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:10:30.160270 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3f9ef40f2bf401e648b20c0b5b472c1735b410e843f94ad0b1104f3d394129\": container with ID starting with 1c3f9ef40f2bf401e648b20c0b5b472c1735b410e843f94ad0b1104f3d394129 not found: ID does not exist" containerID="1c3f9ef40f2bf401e648b20c0b5b472c1735b410e843f94ad0b1104f3d394129" Oct 30 17:10:30.160384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.160302 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3f9ef40f2bf401e648b20c0b5b472c1735b410e843f94ad0b1104f3d394129"} err="failed to get container status \"1c3f9ef40f2bf401e648b20c0b5b472c1735b410e843f94ad0b1104f3d394129\": rpc error: code = NotFound desc = could not find container \"1c3f9ef40f2bf401e648b20c0b5b472c1735b410e843f94ad0b1104f3d394129\": container with ID starting with 1c3f9ef40f2bf401e648b20c0b5b472c1735b410e843f94ad0b1104f3d394129 not found: ID does not exist" Oct 30 17:10:30.160384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.160328 4828 scope.go:122] "RemoveContainer" containerID="ec827807ba76bfab23adc5a6fc8ba8c146b5e59467bbd341f963a894e3a24c23" Oct 30 17:10:30.171822 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.171797 4828 scope.go:122] "RemoveContainer" containerID="ec827807ba76bfab23adc5a6fc8ba8c146b5e59467bbd341f963a894e3a24c23" Oct 30 17:10:30.172313 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:10:30.172288 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec827807ba76bfab23adc5a6fc8ba8c146b5e59467bbd341f963a894e3a24c23\": container with ID starting with ec827807ba76bfab23adc5a6fc8ba8c146b5e59467bbd341f963a894e3a24c23 not found: ID does not exist" containerID="ec827807ba76bfab23adc5a6fc8ba8c146b5e59467bbd341f963a894e3a24c23" Oct 30 17:10:30.172425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.172320 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec827807ba76bfab23adc5a6fc8ba8c146b5e59467bbd341f963a894e3a24c23"} err="failed to get container status \"ec827807ba76bfab23adc5a6fc8ba8c146b5e59467bbd341f963a894e3a24c23\": rpc error: code = NotFound desc = could not find container \"ec827807ba76bfab23adc5a6fc8ba8c146b5e59467bbd341f963a894e3a24c23\": container with ID starting with ec827807ba76bfab23adc5a6fc8ba8c146b5e59467bbd341f963a894e3a24c23 not found: ID does not exist" Oct 30 17:10:30.196998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.196958 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh"] Oct 30 17:10:30.255185 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.241966 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:30.255511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.245306 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-mpwbh"] Oct 30 17:10:30.291948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.290074 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" podStartSLOduration=2.290053373 podStartE2EDuration="2.290053373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:30.287916871 +0000 UTC m=+1407.933427054" watchObservedRunningTime="2026-10-30 17:10:30.290053373 +0000 UTC m=+1407.935563553" Oct 30 17:10:30.386383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.386265 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" podStartSLOduration=2.386247932 podStartE2EDuration="2.386247932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:30.38566761 +0000 UTC m=+1408.031177792" watchObservedRunningTime="2026-10-30 17:10:30.386247932 +0000 UTC m=+1408.031758113" Oct 30 17:10:30.430666 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.430619 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt"] Oct 30 17:10:30.435863 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.435834 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-4s7mt"] Oct 30 17:10:30.465959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.465925 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9"] Oct 30 17:10:30.467578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.467541 4828 cpu_manager.go:414] "containerMap: removing container" podUID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" containerName="istio-proxy" Oct 30 17:10:30.467578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.467573 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" containerName="istio-proxy" Oct 30 17:10:30.467840 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.467594 4828 cpu_manager.go:414] "containerMap: removing container" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerName="istio-proxy" Oct 30 17:10:30.467840 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.467604 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerName="istio-proxy" Oct 30 17:10:30.467840 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.467618 4828 cpu_manager.go:414] "containerMap: removing container" podUID="1fa37ccf-2a69-4083-a6db-b32cd13132a4" containerName="istio-proxy" Oct 30 17:10:30.467840 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.467626 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa37ccf-2a69-4083-a6db-b32cd13132a4" containerName="istio-proxy" Oct 30 17:10:30.467840 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.467832 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" containerName="istio-proxy" Oct 30 17:10:30.468152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.467849 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" containerName="istio-proxy" Oct 30 17:10:30.468152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.467860 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa37ccf-2a69-4083-a6db-b32cd13132a4" containerName="istio-proxy" Oct 30 17:10:30.474681 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.474656 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.476976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.476950 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw"] Oct 30 17:10:30.498160 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.498093 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-432680-openshift-default-dockercfg-nvpvc\"" Oct 30 17:10:30.507762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.506804 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4546e53a-e9f2-4e09-ad63-c9df1bb57570" path="/var/lib/kubelet/pods/4546e53a-e9f2-4e09-ad63-c9df1bb57570/volumes" Oct 30 17:10:30.507762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.507654 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d" path="/var/lib/kubelet/pods/6f7a2ea3-7b3e-4cbe-a78a-da2073164c6d/volumes" Oct 30 17:10:30.508201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.508176 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-trqgw"] Oct 30 17:10:30.517151 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.517115 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9"] Oct 30 17:10:30.554196 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.554161 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2bbc703-7b2d-4c94-8730-63135103703e-istiod-ca-cert\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.554325 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.554232 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-envoy\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.554325 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.554302 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2bbc703-7b2d-4c94-8730-63135103703e-istio-podinfo\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.554449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.554439 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-istio-token\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.554501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.554474 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-credential-socket\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.554552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.554504 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-socket\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.554605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.554589 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-certs\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.554660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.554627 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-data\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.554725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.554709 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hx8k\" (UniqueName: \"kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-kube-api-access-9hx8k\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.584836 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.584785 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:30.656131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.656061 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2bbc703-7b2d-4c94-8730-63135103703e-istio-podinfo\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.656274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.656148 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-istio-token\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.656274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.656187 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-credential-socket\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.656274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.656212 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-socket\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.656424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.656275 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-certs\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.656424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.656299 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-data\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.656424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.656371 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hx8k\" (UniqueName: \"kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-kube-api-access-9hx8k\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.656623 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.656601 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2bbc703-7b2d-4c94-8730-63135103703e-istiod-ca-cert\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.656695 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.656643 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-credential-socket\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.656695 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.656662 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-envoy\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.656805 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.656683 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-socket\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.656805 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.656776 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-certs\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.656886 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.656821 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-data\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.657275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.657255 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2bbc703-7b2d-4c94-8730-63135103703e-istiod-ca-cert\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.658640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.658618 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2bbc703-7b2d-4c94-8730-63135103703e-istio-podinfo\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.658760 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.658744 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-envoy\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.745799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.745757 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:30.762238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.762156 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-istio-token\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.833925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.805722 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hx8k\" (UniqueName: \"kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-kube-api-access-9hx8k\") pod \"e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:30.835791 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:30.835518 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:31.289860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.289836 4828 generic.go:356] "Generic (PLEG): container finished" podID="cdb4b388-a68c-454b-92fc-34a9c1b94a9f" containerID="94e641bcbde48266b4a025e26f0e0b786caa742d962235adef8390b2c1b2c2a6" exitCode=0 Oct 30 17:10:31.290345 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.290324 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" event={"ID":"cdb4b388-a68c-454b-92fc-34a9c1b94a9f","Type":"ContainerDied","Data":"94e641bcbde48266b4a025e26f0e0b786caa742d962235adef8390b2c1b2c2a6"} Oct 30 17:10:31.293342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.293320 4828 generic.go:356] "Generic (PLEG): container finished" podID="4740747b-0ca9-4643-910f-32507d5d0bed" containerID="3c26742939ff7b68081c51a7f4e22c4705382f4281cff7152e1df85645298fac" exitCode=0 Oct 30 17:10:31.293457 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.293443 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" event={"ID":"4740747b-0ca9-4643-910f-32507d5d0bed","Type":"ContainerDied","Data":"3c26742939ff7b68081c51a7f4e22c4705382f4281cff7152e1df85645298fac"} Oct 30 17:10:31.443809 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.443780 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" Oct 30 17:10:31.483180 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.482855 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" Oct 30 17:10:31.565113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.565077 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-plugins-config-volume\") pod \"cdb4b388-a68c-454b-92fc-34a9c1b94a9f\" (UID: \"cdb4b388-a68c-454b-92fc-34a9c1b94a9f\") " Oct 30 17:10:31.565283 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.565228 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4740747b-0ca9-4643-910f-32507d5d0bed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4740747b-0ca9-4643-910f-32507d5d0bed-plugins-config-volume\") pod \"4740747b-0ca9-4643-910f-32507d5d0bed\" (UID: \"4740747b-0ca9-4643-910f-32507d5d0bed\") " Oct 30 17:10:31.565541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.565516 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4740747b-0ca9-4643-910f-32507d5d0bed-kube-api-access-v6rfb\" (UniqueName: \"kubernetes.io/projected/4740747b-0ca9-4643-910f-32507d5d0bed-kube-api-access-v6rfb\") pod \"4740747b-0ca9-4643-910f-32507d5d0bed\" (UID: \"4740747b-0ca9-4643-910f-32507d5d0bed\") " Oct 30 17:10:31.565632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.565610 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-plugins-config-volume" pod "cdb4b388-a68c-454b-92fc-34a9c1b94a9f" (UID: "cdb4b388-a68c-454b-92fc-34a9c1b94a9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:31.565728 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.565629 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-kube-api-access-b5jtd\" (UniqueName: \"kubernetes.io/projected/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-kube-api-access-b5jtd\") pod \"cdb4b388-a68c-454b-92fc-34a9c1b94a9f\" (UID: \"cdb4b388-a68c-454b-92fc-34a9c1b94a9f\") " Oct 30 17:10:31.566148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.566090 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4740747b-0ca9-4643-910f-32507d5d0bed-plugins-config-volume" pod "4740747b-0ca9-4643-910f-32507d5d0bed" (UID: "4740747b-0ca9-4643-910f-32507d5d0bed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:31.566627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.566502 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:31.566627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.566529 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4740747b-0ca9-4643-910f-32507d5d0bed-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:31.568223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.568199 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4740747b-0ca9-4643-910f-32507d5d0bed-kube-api-access-v6rfb" pod "4740747b-0ca9-4643-910f-32507d5d0bed" (UID: "4740747b-0ca9-4643-910f-32507d5d0bed"). InnerVolumeSpecName "kube-api-access-v6rfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:31.568514 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.568484 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-kube-api-access-b5jtd" pod "cdb4b388-a68c-454b-92fc-34a9c1b94a9f" (UID: "cdb4b388-a68c-454b-92fc-34a9c1b94a9f"). InnerVolumeSpecName "kube-api-access-b5jtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:31.600960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.600861 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9"] Oct 30 17:10:31.602778 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:10:31.602709 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bbc703_7b2d_4c94_8730_63135103703e.slice/crio-3eb24d9e6aa0f8ed7da12dd482cb7edf467cb42835bc31d50414c12f4bc7f3ba WatchSource:0}: Error finding container 3eb24d9e6aa0f8ed7da12dd482cb7edf467cb42835bc31d50414c12f4bc7f3ba: Status 404 returned error can't find the container with id 3eb24d9e6aa0f8ed7da12dd482cb7edf467cb42835bc31d50414c12f4bc7f3ba Oct 30 17:10:31.606801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.606761 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:10:31.606909 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.606847 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:10:31.606983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.606891 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:10:31.668590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.668561 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6rfb\" (UniqueName: \"kubernetes.io/projected/4740747b-0ca9-4643-910f-32507d5d0bed-kube-api-access-v6rfb\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:31.668590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:31.668588 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5jtd\" (UniqueName: \"kubernetes.io/projected/cdb4b388-a68c-454b-92fc-34a9c1b94a9f-kube-api-access-b5jtd\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:32.262941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.262647 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:32.380409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.377978 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" event={"ID":"cdb4b388-a68c-454b-92fc-34a9c1b94a9f","Type":"ContainerDied","Data":"78c96f821b090159484a77558dd1911fb11eee24599385e3e70f5fd9d18a674d"} Oct 30 17:10:32.380409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.378061 4828 scope.go:122] "RemoveContainer" containerID="94e641bcbde48266b4a025e26f0e0b786caa742d962235adef8390b2c1b2c2a6" Oct 30 17:10:32.380409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.378233 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw" Oct 30 17:10:32.405716 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.405682 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" Oct 30 17:10:32.406656 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.406611 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" event={"ID":"e2bbc703-7b2d-4c94-8730-63135103703e","Type":"ContainerStarted","Data":"01084f1d8eea4c1d0c123590df00d671ac525ab23d5407a94c53f030547f3d2a"} Oct 30 17:10:32.406739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.406685 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" event={"ID":"e2bbc703-7b2d-4c94-8730-63135103703e","Type":"ContainerStarted","Data":"3eb24d9e6aa0f8ed7da12dd482cb7edf467cb42835bc31d50414c12f4bc7f3ba"} Oct 30 17:10:32.406831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.406813 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6" event={"ID":"4740747b-0ca9-4643-910f-32507d5d0bed","Type":"ContainerDied","Data":"0db9403765edcb5b595380c4f8b5c2c1598a4932349d9b2e3472d552ce8439ff"} Oct 30 17:10:32.420610 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.420586 4828 scope.go:122] "RemoveContainer" containerID="3c26742939ff7b68081c51a7f4e22c4705382f4281cff7152e1df85645298fac" Oct 30 17:10:32.463766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.463703 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" podStartSLOduration=2.463689137 podStartE2EDuration="2.463689137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:32.458451288 +0000 UTC m=+1410.103961468" watchObservedRunningTime="2026-10-30 17:10:32.463689137 +0000 UTC m=+1410.109199351" Oct 30 17:10:32.497679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.497642 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa37ccf-2a69-4083-a6db-b32cd13132a4" path="/var/lib/kubelet/pods/1fa37ccf-2a69-4083-a6db-b32cd13132a4/volumes" Oct 30 17:10:32.516800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.516758 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6"] Oct 30 17:10:32.521753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.521711 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ebeb6e1-766b45f899-828s6"] Oct 30 17:10:32.556440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.556406 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw"] Oct 30 17:10:32.561344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.561313 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-bldzw"] Oct 30 17:10:32.841667 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.841633 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:32.847836 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:32.847809 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:33.418883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:33.418844 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:33.420042 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:33.420020 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:10:34.496797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:34.496761 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4740747b-0ca9-4643-910f-32507d5d0bed" path="/var/lib/kubelet/pods/4740747b-0ca9-4643-910f-32507d5d0bed/volumes" Oct 30 17:10:34.497394 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:34.497376 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdb4b388-a68c-454b-92fc-34a9c1b94a9f" path="/var/lib/kubelet/pods/cdb4b388-a68c-454b-92fc-34a9c1b94a9f/volumes" Oct 30 17:10:35.668656 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:35.668617 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:35.865924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:35.865356 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:36.508859 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.508789 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g"] Oct 30 17:10:36.510211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.510133 4828 cpu_manager.go:414] "containerMap: removing container" podUID="4740747b-0ca9-4643-910f-32507d5d0bed" containerName="payload-pre-processing" Oct 30 17:10:36.510211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.510158 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4740747b-0ca9-4643-910f-32507d5d0bed" containerName="payload-pre-processing" Oct 30 17:10:36.510211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.510189 4828 cpu_manager.go:414] "containerMap: removing container" podUID="cdb4b388-a68c-454b-92fc-34a9c1b94a9f" containerName="payload-processing" Oct 30 17:10:36.510211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.510197 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdb4b388-a68c-454b-92fc-34a9c1b94a9f" containerName="payload-processing" Oct 30 17:10:36.510593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.510427 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="4740747b-0ca9-4643-910f-32507d5d0bed" containerName="payload-pre-processing" Oct 30 17:10:36.510593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.510449 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdb4b388-a68c-454b-92fc-34a9c1b94a9f" containerName="payload-processing" Oct 30 17:10:36.518775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.518741 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:36.519277 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.519257 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g"] Oct 30 17:10:36.534853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.534826 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-362bbf-serving-cert\"" Oct 30 17:10:36.621481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.621446 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:36.681928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.680525 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2459de27-81ba-4d49-886e-8629d8cdbf60-openshift-service-ca\") pod \"maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:36.681928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.680734 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hscv\" (UniqueName: \"kubernetes.io/projected/2459de27-81ba-4d49-886e-8629d8cdbf60-kube-api-access-2hscv\") pod \"maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:36.681928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.681015 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-maas-api-tls\") pod \"maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:36.681928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.681067 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-metrics-certs\") pod \"maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:36.788971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.786714 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-metrics-certs\") pod \"maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:36.788971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.787384 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2459de27-81ba-4d49-886e-8629d8cdbf60-openshift-service-ca\") pod \"maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:36.788971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.787567 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hscv\" (UniqueName: \"kubernetes.io/projected/2459de27-81ba-4d49-886e-8629d8cdbf60-kube-api-access-2hscv\") pod \"maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:36.788971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.787844 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-maas-api-tls\") pod \"maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:36.789300 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.789159 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2459de27-81ba-4d49-886e-8629d8cdbf60-openshift-service-ca\") pod \"maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:36.808584 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.806853 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-metrics-certs\") pod \"maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:36.811670 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.811635 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-maas-api-tls\") pod \"maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:36.814833 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.814801 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hscv\" (UniqueName: \"kubernetes.io/projected/2459de27-81ba-4d49-886e-8629d8cdbf60-kube-api-access-2hscv\") pod \"maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:36.862920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.861825 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:36.862920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:36.862384 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:37.009979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.009920 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:37.211008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.207599 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g"] Oct 30 17:10:37.211008 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:10:37.207684 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2459de27_81ba_4d49_886e_8629d8cdbf60.slice/crio-3ba1622e437b1aee9edd841ec70fe00080e5a1a86100a6f97caf1ec839b88cb8 WatchSource:0}: Error finding container 3ba1622e437b1aee9edd841ec70fe00080e5a1a86100a6f97caf1ec839b88cb8: Status 404 returned error can't find the container with id 3ba1622e437b1aee9edd841ec70fe00080e5a1a86100a6f97caf1ec839b88cb8 Oct 30 17:10:37.233031 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.233000 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:37.371123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.371088 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk"] Oct 30 17:10:37.377551 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.377523 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" Oct 30 17:10:37.398473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.397688 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-432680\"" Oct 30 17:10:37.400138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.400115 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk"] Oct 30 17:10:37.402886 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.402833 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:37.405815 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.405791 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-432680-dockercfg-2dt24\"" Oct 30 17:10:37.612949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.612869 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk\" (UID: \"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" Oct 30 17:10:37.613175 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.612957 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8mv\" (UniqueName: \"kubernetes.io/projected/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-kube-api-access-6v8mv\") pod \"payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk\" (UID: \"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" Oct 30 17:10:37.725027 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.723234 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk\" (UID: \"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" Oct 30 17:10:37.725027 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.723308 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8mv\" (UniqueName: \"kubernetes.io/projected/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-kube-api-access-6v8mv\") pod \"payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk\" (UID: \"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" Oct 30 17:10:37.733985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.730642 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk\" (UID: \"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" Oct 30 17:10:37.967500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.967404 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" event={"ID":"2459de27-81ba-4d49-886e-8629d8cdbf60","Type":"ContainerStarted","Data":"3ba1622e437b1aee9edd841ec70fe00080e5a1a86100a6f97caf1ec839b88cb8"} Oct 30 17:10:37.988798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.988739 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8mv\" (UniqueName: \"kubernetes.io/projected/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-kube-api-access-6v8mv\") pod \"payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk\" (UID: \"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" Oct 30 17:10:37.995277 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:37.995246 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr"] Oct 30 17:10:38.001824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.001784 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" Oct 30 17:10:38.002125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.002109 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr"] Oct 30 17:10:38.002285 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.002262 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" Oct 30 17:10:38.028167 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:10:38.028135 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:10:38.067989 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.067951 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z798x\" (UniqueName: \"kubernetes.io/projected/a8eb2ac7-e218-4729-bc72-578ee763e55a-kube-api-access-z798x\") pod \"payload-processing-e2e-api-a-432680-85c94bccd8-gpspr\" (UID: \"a8eb2ac7-e218-4729-bc72-578ee763e55a\") " pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" Oct 30 17:10:38.068928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.068257 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8eb2ac7-e218-4729-bc72-578ee763e55a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-432680-85c94bccd8-gpspr\" (UID: \"a8eb2ac7-e218-4729-bc72-578ee763e55a\") " pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" Oct 30 17:10:38.169769 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.169726 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z798x\" (UniqueName: \"kubernetes.io/projected/a8eb2ac7-e218-4729-bc72-578ee763e55a-kube-api-access-z798x\") pod \"payload-processing-e2e-api-a-432680-85c94bccd8-gpspr\" (UID: \"a8eb2ac7-e218-4729-bc72-578ee763e55a\") " pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" Oct 30 17:10:38.169986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.169965 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8eb2ac7-e218-4729-bc72-578ee763e55a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-432680-85c94bccd8-gpspr\" (UID: \"a8eb2ac7-e218-4729-bc72-578ee763e55a\") " pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" Oct 30 17:10:38.171265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.171243 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8eb2ac7-e218-4729-bc72-578ee763e55a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-432680-85c94bccd8-gpspr\" (UID: \"a8eb2ac7-e218-4729-bc72-578ee763e55a\") " pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" Oct 30 17:10:38.204701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.204666 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z798x\" (UniqueName: \"kubernetes.io/projected/a8eb2ac7-e218-4729-bc72-578ee763e55a-kube-api-access-z798x\") pod \"payload-processing-e2e-api-a-432680-85c94bccd8-gpspr\" (UID: \"a8eb2ac7-e218-4729-bc72-578ee763e55a\") " pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" Oct 30 17:10:38.347924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.337334 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" Oct 30 17:10:38.472293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.472065 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk"] Oct 30 17:10:38.652592 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.642267 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc"] Oct 30 17:10:38.652592 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.642503 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" podUID="d2208a20-2b4b-4cbd-a471-c39873e21f09" containerName="maas-api" containerID="cri-o://7ac013e39ee79b687edb7c5d0c53cc97929c08791191aaa6928a92d7a5ed746f" gracePeriod=30 Oct 30 17:10:38.747805 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:38.747756 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:39.467313 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:39.456470 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr"] Oct 30 17:10:39.515077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:39.508438 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" event={"ID":"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a","Type":"ContainerStarted","Data":"e2843c27811da4f379b7c02119bd436dac1028fa2f5bc7bb10207fdab54e2e97"} Oct 30 17:10:39.515077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:39.508507 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" event={"ID":"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a","Type":"ContainerStarted","Data":"7ace575053b271d2ca222c6a01a8255e7ce9c917dd7b2225829068d7f38b93fc"} Oct 30 17:10:39.515077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:39.509628 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" Oct 30 17:10:39.742014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:39.737328 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:39.829890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:39.817389 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" event={"ID":"2459de27-81ba-4d49-886e-8629d8cdbf60","Type":"ContainerStarted","Data":"5a80a93444f37aa6bad98f9f0dbe0b59e797cf81ba9ce1d4ad58a5ad27441a86"} Oct 30 17:10:39.870739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:39.870697 4828 generic.go:356] "Generic (PLEG): container finished" podID="d2208a20-2b4b-4cbd-a471-c39873e21f09" containerID="7ac013e39ee79b687edb7c5d0c53cc97929c08791191aaa6928a92d7a5ed746f" exitCode=0 Oct 30 17:10:39.871116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:39.871075 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" event={"ID":"d2208a20-2b4b-4cbd-a471-c39873e21f09","Type":"ContainerDied","Data":"7ac013e39ee79b687edb7c5d0c53cc97929c08791191aaa6928a92d7a5ed746f"} Oct 30 17:10:39.871628 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:39.871606 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:39.961889 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:39.925637 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:39.962080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:39.941268 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" podStartSLOduration=2.941248587 podStartE2EDuration="2.941248587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:39.734989675 +0000 UTC m=+1417.380499854" watchObservedRunningTime="2026-10-30 17:10:39.941248587 +0000 UTC m=+1417.586758771" Oct 30 17:10:39.979270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:39.979207 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" podStartSLOduration=3.19667092 podStartE2EDuration="3.979187113s" totalImagesPullingTime="782.516193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:39.939205489 +0000 UTC m=+1417.584715669" watchObservedRunningTime="2026-10-30 17:10:39.979187113 +0000 UTC m=+1417.624697298" Oct 30 17:10:39.984842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:39.984820 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:10:40.134783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.134744 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2208a20-2b4b-4cbd-a471-c39873e21f09-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2208a20-2b4b-4cbd-a471-c39873e21f09-openshift-service-ca\") pod \"d2208a20-2b4b-4cbd-a471-c39873e21f09\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " Oct 30 17:10:40.135030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.134830 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-maas-api-tls\") pod \"d2208a20-2b4b-4cbd-a471-c39873e21f09\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " Oct 30 17:10:40.135030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.135014 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-metrics-certs\") pod \"d2208a20-2b4b-4cbd-a471-c39873e21f09\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " Oct 30 17:10:40.135193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.135177 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2208a20-2b4b-4cbd-a471-c39873e21f09-openshift-service-ca" pod "d2208a20-2b4b-4cbd-a471-c39873e21f09" (UID: "d2208a20-2b4b-4cbd-a471-c39873e21f09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:40.135250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.135233 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2208a20-2b4b-4cbd-a471-c39873e21f09-kube-api-access-22tjl\" (UniqueName: \"kubernetes.io/projected/d2208a20-2b4b-4cbd-a471-c39873e21f09-kube-api-access-22tjl\") pod \"d2208a20-2b4b-4cbd-a471-c39873e21f09\" (UID: \"d2208a20-2b4b-4cbd-a471-c39873e21f09\") " Oct 30 17:10:40.135943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.135892 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2208a20-2b4b-4cbd-a471-c39873e21f09-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:40.137398 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.137369 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-metrics-certs" pod "d2208a20-2b4b-4cbd-a471-c39873e21f09" (UID: "d2208a20-2b4b-4cbd-a471-c39873e21f09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:40.137562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.137544 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-maas-api-tls" pod "d2208a20-2b4b-4cbd-a471-c39873e21f09" (UID: "d2208a20-2b4b-4cbd-a471-c39873e21f09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:40.137621 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.137566 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2208a20-2b4b-4cbd-a471-c39873e21f09-kube-api-access-22tjl" pod "d2208a20-2b4b-4cbd-a471-c39873e21f09" (UID: "d2208a20-2b4b-4cbd-a471-c39873e21f09"). InnerVolumeSpecName "kube-api-access-22tjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:40.196082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.192094 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:40.238188 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.238139 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22tjl\" (UniqueName: \"kubernetes.io/projected/d2208a20-2b4b-4cbd-a471-c39873e21f09-kube-api-access-22tjl\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:40.238188 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.238175 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:40.238188 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.238190 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2208a20-2b4b-4cbd-a471-c39873e21f09-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:40.417650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.417562 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:40.881844 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.881805 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" event={"ID":"a8eb2ac7-e218-4729-bc72-578ee763e55a","Type":"ContainerStarted","Data":"8a167aba21297860709b3717138489328ac6d0f525947258eae4b8cfc012fad7"} Oct 30 17:10:40.882362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.881858 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" event={"ID":"a8eb2ac7-e218-4729-bc72-578ee763e55a","Type":"ContainerStarted","Data":"81cceea6e5d9d608080271a437ca37aa861bcec94576fddfea2206368bdcb94c"} Oct 30 17:10:40.882362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.881948 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" Oct 30 17:10:40.883572 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.883547 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" Oct 30 17:10:40.883706 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.883576 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc" event={"ID":"d2208a20-2b4b-4cbd-a471-c39873e21f09","Type":"ContainerDied","Data":"93b090036763424cb2aefbc71cb458bedc947c9dc1ef325c362e69cc3d897a61"} Oct 30 17:10:40.883706 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.883631 4828 scope.go:122] "RemoveContainer" containerID="7ac013e39ee79b687edb7c5d0c53cc97929c08791191aaa6928a92d7a5ed746f" Oct 30 17:10:40.957011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:40.956943 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" podStartSLOduration=3.956885568 podStartE2EDuration="3.956885568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:40.956819323 +0000 UTC m=+1418.602329502" watchObservedRunningTime="2026-10-30 17:10:40.956885568 +0000 UTC m=+1418.602395748" Oct 30 17:10:41.009847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:41.009811 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc"] Oct 30 17:10:41.015066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:41.015039 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c425a3e3-5df9746659-j66pc"] Oct 30 17:10:41.301595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:41.301567 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" Oct 30 17:10:41.301780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:41.301640 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" Oct 30 17:10:42.043486 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:42.040894 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:42.496481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:42.496408 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2208a20-2b4b-4cbd-a471-c39873e21f09" path="/var/lib/kubelet/pods/d2208a20-2b4b-4cbd-a471-c39873e21f09/volumes" Oct 30 17:10:43.822127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:43.822079 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:45.571927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:45.567880 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:46.900748 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:46.900716 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:10:46.993955 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:46.985563 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:47.296199 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:47.289634 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:47.728625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:47.706827 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l"] Oct 30 17:10:47.729560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:47.729494 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerName="istio-proxy" containerID="cri-o://fcf78c4702e30a53e77433daa0aa97f7cb82ead72aaa47692a29f8bef5c4b477" gracePeriod=30 Oct 30 17:10:47.757635 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:47.755331 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:48.276003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:48.275961 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:48.459742 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:48.459691 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:49.460131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:49.460068 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:49.719980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:49.719157 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:49.916926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:49.903412 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:50.128928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:50.128879 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:50.279865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:50.279822 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:51.275240 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:51.275201 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:51.898214 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:51.898183 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" Oct 30 17:10:51.898424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:51.898275 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" Oct 30 17:10:52.960642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:52.960600 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8"] Oct 30 17:10:52.961475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:52.961452 4828 cpu_manager.go:414] "containerMap: removing container" podUID="d2208a20-2b4b-4cbd-a471-c39873e21f09" containerName="maas-api" Oct 30 17:10:52.961475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:52.961470 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2208a20-2b4b-4cbd-a471-c39873e21f09" containerName="maas-api" Oct 30 17:10:52.961649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:52.961633 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2208a20-2b4b-4cbd-a471-c39873e21f09" containerName="maas-api" Oct 30 17:10:52.965366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:52.965345 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:52.995151 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:52.991208 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c4c2e4-serving-cert\"" Oct 30 17:10:52.995151 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:52.994409 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8"] Oct 30 17:10:53.034838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.034813 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-maas-api-tls\") pod \"maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:53.034987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.034888 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znlw6\" (UniqueName: \"kubernetes.io/projected/77de8b26-73dc-4658-aec4-4e582950e6de-kube-api-access-znlw6\") pod \"maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:53.034987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.034961 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-metrics-certs\") pod \"maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:53.035081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.035012 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77de8b26-73dc-4658-aec4-4e582950e6de-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:53.137022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.136990 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-maas-api-tls\") pod \"maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:53.137225 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.137103 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlw6\" (UniqueName: \"kubernetes.io/projected/77de8b26-73dc-4658-aec4-4e582950e6de-kube-api-access-znlw6\") pod \"maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:53.137225 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.137154 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-metrics-certs\") pod \"maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:53.137347 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.137242 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77de8b26-73dc-4658-aec4-4e582950e6de-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:53.140357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.139070 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77de8b26-73dc-4658-aec4-4e582950e6de-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:53.142660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.139388 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:53.142782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.140731 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-maas-api-tls\") pod \"maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:53.142782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.141141 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-metrics-certs\") pod \"maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:53.250479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.250184 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlw6\" (UniqueName: \"kubernetes.io/projected/77de8b26-73dc-4658-aec4-4e582950e6de-kube-api-access-znlw6\") pod \"maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:53.415346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.415087 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:53.432017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.430751 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:53.438566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.438469 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:10:53.476865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.476825 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-envoy\") pod \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " Oct 30 17:10:53.477066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.476890 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-data\") pod \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " Oct 30 17:10:53.477066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.476991 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-certs\") pod \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " Oct 30 17:10:53.477199 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.477095 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-socket\") pod \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " Oct 30 17:10:53.477426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.477282 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-kube-api-access-447c9\" (UniqueName: \"kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-kube-api-access-447c9\") pod \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " Oct 30 17:10:53.477426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.477363 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-podinfo\") pod \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " Oct 30 17:10:53.477426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.477413 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istiod-ca-cert\") pod \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " Oct 30 17:10:53.477664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.477440 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-data" pod "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" (UID: "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:53.477664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.477457 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-token\" (UniqueName: \"kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-token\") pod \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " Oct 30 17:10:53.477784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.477669 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-certs" pod "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" (UID: "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:53.477784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.477736 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-credential-socket\") pod \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\" (UID: \"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4\") " Oct 30 17:10:53.480848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.478453 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-socket" pod "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" (UID: "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:53.480848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.478816 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-credential-socket" pod "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" (UID: "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:53.480848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.478945 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:53.480848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.478964 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:53.480848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.478978 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:53.480848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.479490 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istiod-ca-cert" pod "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" (UID: "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:53.482088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.482063 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-envoy" pod "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" (UID: "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:53.483948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.483924 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-kube-api-access-447c9" pod "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" (UID: "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4"). InnerVolumeSpecName "kube-api-access-447c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:53.484513 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.484486 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-podinfo" pod "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" (UID: "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:10:53.484851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.484829 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-token" pod "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" (UID: "be4bd50f-93c7-4ba8-8e58-ef48c1a178a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:53.580943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.580866 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:53.581133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.581003 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:53.581133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.581024 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:53.581133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.581039 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:53.581133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.581054 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:53.581133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.581072 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-447c9\" (UniqueName: \"kubernetes.io/projected/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4-kube-api-access-447c9\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:10:53.720790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.720752 4828 generic.go:356] "Generic (PLEG): container finished" podID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerID="fcf78c4702e30a53e77433daa0aa97f7cb82ead72aaa47692a29f8bef5c4b477" exitCode=0 Oct 30 17:10:53.720951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.720812 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" Oct 30 17:10:53.720951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.720830 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" event={"ID":"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4","Type":"ContainerDied","Data":"fcf78c4702e30a53e77433daa0aa97f7cb82ead72aaa47692a29f8bef5c4b477"} Oct 30 17:10:53.720951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.720890 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l" event={"ID":"be4bd50f-93c7-4ba8-8e58-ef48c1a178a4","Type":"ContainerDied","Data":"99a43e5819b7d06b693187e289e13fb971bb3997bfe454301941337f061036f9"} Oct 30 17:10:53.720951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.720925 4828 scope.go:122] "RemoveContainer" containerID="fcf78c4702e30a53e77433daa0aa97f7cb82ead72aaa47692a29f8bef5c4b477" Oct 30 17:10:53.733058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.733037 4828 scope.go:122] "RemoveContainer" containerID="6c58eee8fedf51587c1839a7d8587d6fabf97d5aa33acab6e59983f0bfed6ce0" Oct 30 17:10:53.743839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.743806 4828 scope.go:122] "RemoveContainer" containerID="fcf78c4702e30a53e77433daa0aa97f7cb82ead72aaa47692a29f8bef5c4b477" Oct 30 17:10:53.744153 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:10:53.744133 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf78c4702e30a53e77433daa0aa97f7cb82ead72aaa47692a29f8bef5c4b477\": container with ID starting with fcf78c4702e30a53e77433daa0aa97f7cb82ead72aaa47692a29f8bef5c4b477 not found: ID does not exist" containerID="fcf78c4702e30a53e77433daa0aa97f7cb82ead72aaa47692a29f8bef5c4b477" Oct 30 17:10:53.744228 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.744163 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf78c4702e30a53e77433daa0aa97f7cb82ead72aaa47692a29f8bef5c4b477"} err="failed to get container status \"fcf78c4702e30a53e77433daa0aa97f7cb82ead72aaa47692a29f8bef5c4b477\": rpc error: code = NotFound desc = could not find container \"fcf78c4702e30a53e77433daa0aa97f7cb82ead72aaa47692a29f8bef5c4b477\": container with ID starting with fcf78c4702e30a53e77433daa0aa97f7cb82ead72aaa47692a29f8bef5c4b477 not found: ID does not exist" Oct 30 17:10:53.744228 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.744186 4828 scope.go:122] "RemoveContainer" containerID="6c58eee8fedf51587c1839a7d8587d6fabf97d5aa33acab6e59983f0bfed6ce0" Oct 30 17:10:53.744509 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:10:53.744470 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c58eee8fedf51587c1839a7d8587d6fabf97d5aa33acab6e59983f0bfed6ce0\": container with ID starting with 6c58eee8fedf51587c1839a7d8587d6fabf97d5aa33acab6e59983f0bfed6ce0 not found: ID does not exist" containerID="6c58eee8fedf51587c1839a7d8587d6fabf97d5aa33acab6e59983f0bfed6ce0" Oct 30 17:10:53.744576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.744507 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c58eee8fedf51587c1839a7d8587d6fabf97d5aa33acab6e59983f0bfed6ce0"} err="failed to get container status \"6c58eee8fedf51587c1839a7d8587d6fabf97d5aa33acab6e59983f0bfed6ce0\": rpc error: code = NotFound desc = could not find container \"6c58eee8fedf51587c1839a7d8587d6fabf97d5aa33acab6e59983f0bfed6ce0\": container with ID starting with 6c58eee8fedf51587c1839a7d8587d6fabf97d5aa33acab6e59983f0bfed6ce0 not found: ID does not exist" Oct 30 17:10:53.750371 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:10:53.750345 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77de8b26_73dc_4658_aec4_4e582950e6de.slice/crio-074636cdb0243501b42d00fc59ab8640ec90cebc6b9ed83946b42381eba9b7bf WatchSource:0}: Error finding container 074636cdb0243501b42d00fc59ab8640ec90cebc6b9ed83946b42381eba9b7bf: Status 404 returned error can't find the container with id 074636cdb0243501b42d00fc59ab8640ec90cebc6b9ed83946b42381eba9b7bf Oct 30 17:10:53.752770 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.752742 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8"] Oct 30 17:10:53.752871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.752757 4828 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:10:53.781391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.781349 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l"] Oct 30 17:10:53.787525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:53.787487 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c425a3e3-openshift-default-d8cc475d9-9978l"] Oct 30 17:10:54.549159 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:54.548281 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" path="/var/lib/kubelet/pods/be4bd50f-93c7-4ba8-8e58-ef48c1a178a4/volumes" Oct 30 17:10:54.752204 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:54.752064 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" event={"ID":"77de8b26-73dc-4658-aec4-4e582950e6de","Type":"ContainerStarted","Data":"860865cebbbdde6bf95846b06bfc60de2b5dff37ee23367b1710fc067ef76af7"} Oct 30 17:10:54.752204 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:54.752192 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" event={"ID":"77de8b26-73dc-4658-aec4-4e582950e6de","Type":"ContainerStarted","Data":"074636cdb0243501b42d00fc59ab8640ec90cebc6b9ed83946b42381eba9b7bf"} Oct 30 17:10:54.752432 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:54.752351 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:10:54.799659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:54.799528 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" podStartSLOduration=2.429424906 podStartE2EDuration="2.799510858s" totalImagesPullingTime="370.085952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:54.798042449 +0000 UTC m=+1432.443552628" watchObservedRunningTime="2026-10-30 17:10:54.799510858 +0000 UTC m=+1432.445021038" Oct 30 17:10:55.091991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:55.091200 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:56.073031 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:56.072989 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:56.373920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:56.373819 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:56.701248 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:56.701147 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:58.260394 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.260360 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz"] Oct 30 17:10:58.261274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.261254 4828 cpu_manager.go:414] "containerMap: removing container" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerName="istio-proxy" Oct 30 17:10:58.261274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.261274 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerName="istio-proxy" Oct 30 17:10:58.261417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.261295 4828 cpu_manager.go:414] "containerMap: removing container" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerName="istio-proxy" Oct 30 17:10:58.261417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.261303 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerName="istio-proxy" Oct 30 17:10:58.261526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.261512 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerName="istio-proxy" Oct 30 17:10:58.261562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.261532 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="be4bd50f-93c7-4ba8-8e58-ef48c1a178a4" containerName="istio-proxy" Oct 30 17:10:58.266937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.266885 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:58.287727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.287696 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz"] Oct 30 17:10:58.289772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.289750 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-8681b5-serving-cert\"" Oct 30 17:10:58.321509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.321471 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:58.321719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.321692 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:58.321838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.321818 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2c4g\" (UniqueName: \"kubernetes.io/projected/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-kube-api-access-g2c4g\") pod \"maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:58.322011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.321992 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:58.380616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.380577 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:58.424177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.424142 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2c4g\" (UniqueName: \"kubernetes.io/projected/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-kube-api-access-g2c4g\") pod \"maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:58.424405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.424327 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:58.424470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.424444 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:58.424593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.424574 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:58.425403 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.425318 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:58.426934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.426894 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:58.427178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.427156 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:58.449413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.449385 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2c4g\" (UniqueName: \"kubernetes.io/projected/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-kube-api-access-g2c4g\") pod \"maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:58.587063 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.586972 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:10:58.647630 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:58.647593 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:59.094195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:59.089384 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz"] Oct 30 17:10:59.884653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:59.884606 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" event={"ID":"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9","Type":"ContainerStarted","Data":"4a2b751e068e08f0279d080ba767bb54d72c80dfdfda993a6cafd3e15bcc6185"} Oct 30 17:10:59.885154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:59.884682 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" event={"ID":"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9","Type":"ContainerStarted","Data":"76403eb0bd5ceb3c4c085a3e86b279b981cbc7672ad3e0a5614fbf51d9c1cca5"} Oct 30 17:10:59.885154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:59.884734 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:10:59.933864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:10:59.933806 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" podStartSLOduration=1.581081607 podStartE2EDuration="1.933787629s" totalImagesPullingTime="352.706022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:59.929202906 +0000 UTC m=+1437.574713087" watchObservedRunningTime="2026-10-30 17:10:59.933787629 +0000 UTC m=+1437.579297809" Oct 30 17:11:00.201177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:00.201137 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:00.769956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:00.769928 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:11:01.673373 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:01.673335 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:01.854926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:01.844055 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:02.412926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:02.407572 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:02.850970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:02.848568 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:03.145266 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:03.142334 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:03.315925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:03.310516 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:03.459006 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:03.458888 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:05.164927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:05.155837 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:05.906139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:05.906104 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:11:06.915726 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:06.915690 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:07.070402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:07.070367 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:07.916787 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:11:07.916741 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:11:08.495517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:08.495482 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:08.661803 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:08.661562 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:09.398843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:09.398802 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d"] Oct 30 17:11:09.399722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:09.399686 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" podUID="5aa9acde-75ca-4422-ae94-ab6dbd630313" containerName="istio-proxy" containerID="cri-o://64f0496dd22662f2d854bf8970100ba8b29f9f392bb15dda6de1245a0e4e2252" gracePeriod=30 Oct 30 17:11:09.446725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:09.446444 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t"] Oct 30 17:11:09.447105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:09.447050 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerName="istio-proxy" containerID="cri-o://42f48b1c105aaa2a258b64c48051c8836d3005379547f24d41ff4033a2e7ee50" gracePeriod=30 Oct 30 17:11:09.475588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:09.475547 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl"] Oct 30 17:11:09.485053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:09.485019 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk"] Oct 30 17:11:09.486319 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:09.486250 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" podUID="e88601bd-82a5-47c7-be65-67915627db91" containerName="istio-proxy" containerID="cri-o://4c747700874f5ff6f4905b4afca2cfd882305632ec2cd42befdb1b3dc7da4f07" gracePeriod=30 Oct 30 17:11:09.487351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:09.486674 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" containerID="cri-o://3b8dc5a708fbd6075978e0d343b6ef59271d95877bd04a983496b18a0cacfab8" gracePeriod=30 Oct 30 17:11:10.456486 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:10.456448 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:10.613045 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:10.612806 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:10.897312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:10.897270 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:11:10.897483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:10.897361 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:11:11.220417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.220330 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:11.353533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.353495 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5"] Oct 30 17:11:11.360893 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.360868 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" Oct 30 17:11:11.398960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.397435 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7ebeb6e1\"" Oct 30 17:11:11.398960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.398213 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7ebeb6e1-dockercfg-42m7h\"" Oct 30 17:11:11.400361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.400321 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af96ea0-8c31-4d19-ac72-6d7083e590b2-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5\" (UID: \"5af96ea0-8c31-4d19-ac72-6d7083e590b2\") " pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" Oct 30 17:11:11.400499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.400388 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kprwh\" (UniqueName: \"kubernetes.io/projected/5af96ea0-8c31-4d19-ac72-6d7083e590b2-kube-api-access-kprwh\") pod \"payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5\" (UID: \"5af96ea0-8c31-4d19-ac72-6d7083e590b2\") " pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" Oct 30 17:11:11.402338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.402309 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5"] Oct 30 17:11:11.407439 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.407409 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:11.591018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.518841 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af96ea0-8c31-4d19-ac72-6d7083e590b2-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5\" (UID: \"5af96ea0-8c31-4d19-ac72-6d7083e590b2\") " pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" Oct 30 17:11:11.591018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.518918 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kprwh\" (UniqueName: \"kubernetes.io/projected/5af96ea0-8c31-4d19-ac72-6d7083e590b2-kube-api-access-kprwh\") pod \"payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5\" (UID: \"5af96ea0-8c31-4d19-ac72-6d7083e590b2\") " pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" Oct 30 17:11:11.591018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.537501 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af96ea0-8c31-4d19-ac72-6d7083e590b2-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5\" (UID: \"5af96ea0-8c31-4d19-ac72-6d7083e590b2\") " pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" Oct 30 17:11:11.652927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.651684 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kprwh\" (UniqueName: \"kubernetes.io/projected/5af96ea0-8c31-4d19-ac72-6d7083e590b2-kube-api-access-kprwh\") pod \"payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5\" (UID: \"5af96ea0-8c31-4d19-ac72-6d7083e590b2\") " pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" Oct 30 17:11:11.689534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:11.689250 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" Oct 30 17:11:12.333040 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:11:12.333007 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af96ea0_8c31_4d19_ac72_6d7083e590b2.slice/crio-2530a480f29d4a6a8839baefdff343601149cffbe112f33561ea261c6c7f553f WatchSource:0}: Error finding container 2530a480f29d4a6a8839baefdff343601149cffbe112f33561ea261c6c7f553f: Status 404 returned error can't find the container with id 2530a480f29d4a6a8839baefdff343601149cffbe112f33561ea261c6c7f553f Oct 30 17:11:12.335307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:12.335276 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5"] Oct 30 17:11:12.406102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:12.405630 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" event={"ID":"5af96ea0-8c31-4d19-ac72-6d7083e590b2","Type":"ContainerStarted","Data":"2530a480f29d4a6a8839baefdff343601149cffbe112f33561ea261c6c7f553f"} Oct 30 17:11:12.441496 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:12.441462 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:12.608440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:12.604179 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:13.420668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:13.420635 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" event={"ID":"5af96ea0-8c31-4d19-ac72-6d7083e590b2","Type":"ContainerStarted","Data":"c423355c9d119e2771a459d44b30764e56b82fd410a230c471198340321ded43"} Oct 30 17:11:13.420833 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:13.420742 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" Oct 30 17:11:13.471111 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:13.471035 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" podStartSLOduration=2.471015715 podStartE2EDuration="2.471015715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:13.469268983 +0000 UTC m=+1451.114779164" watchObservedRunningTime="2026-10-30 17:11:13.471015715 +0000 UTC m=+1451.116525895" Oct 30 17:11:14.569652 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:14.569496 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:15.035936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.035894 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:11:15.079084 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.078991 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:11:15.084365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.084270 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:11:15.183700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.183526 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-data\") pod \"5cf16d96-e3f2-4982-92dc-dc1188d30890\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " Oct 30 17:11:15.183850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.183812 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-certs\") pod \"5aa9acde-75ca-4422-ae94-ab6dbd630313\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " Oct 30 17:11:15.183850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.183835 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-credential-socket\") pod \"5cf16d96-e3f2-4982-92dc-dc1188d30890\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " Oct 30 17:11:15.184031 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.183854 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-data\") pod \"5aa9acde-75ca-4422-ae94-ab6dbd630313\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " Oct 30 17:11:15.184031 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.183915 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-socket\") pod \"5cf16d96-e3f2-4982-92dc-dc1188d30890\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " Oct 30 17:11:15.184031 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.183949 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-token\" (UniqueName: \"kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-token\") pod \"5aa9acde-75ca-4422-ae94-ab6dbd630313\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " Oct 30 17:11:15.184218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.184056 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-token\" (UniqueName: \"kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-token\") pod \"f3df5215-60e4-4c09-904e-b891da8ee19f\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " Oct 30 17:11:15.184218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.184088 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-podinfo\") pod \"f3df5215-60e4-4c09-904e-b891da8ee19f\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " Oct 30 17:11:15.184218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.184117 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-certs\") pod \"5cf16d96-e3f2-4982-92dc-dc1188d30890\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " Oct 30 17:11:15.184218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.184154 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-certs" pod "5aa9acde-75ca-4422-ae94-ab6dbd630313" (UID: "5aa9acde-75ca-4422-ae94-ab6dbd630313"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.184218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.184162 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cf16d96-e3f2-4982-92dc-dc1188d30890-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf16d96-e3f2-4982-92dc-dc1188d30890-istiod-ca-cert\") pod \"5cf16d96-e3f2-4982-92dc-dc1188d30890\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " Oct 30 17:11:15.184218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.184215 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-podinfo\") pod \"5aa9acde-75ca-4422-ae94-ab6dbd630313\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " Oct 30 17:11:15.184529 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.184261 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-podinfo\") pod \"5cf16d96-e3f2-4982-92dc-dc1188d30890\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " Oct 30 17:11:15.184529 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.184339 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3df5215-60e4-4c09-904e-b891da8ee19f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3df5215-60e4-4c09-904e-b891da8ee19f-istiod-ca-cert\") pod \"f3df5215-60e4-4c09-904e-b891da8ee19f\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " Oct 30 17:11:15.184529 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.184381 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-envoy\") pod \"5aa9acde-75ca-4422-ae94-ab6dbd630313\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " Oct 30 17:11:15.184529 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.184408 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-data\") pod \"f3df5215-60e4-4c09-904e-b891da8ee19f\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " Oct 30 17:11:15.184529 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.184463 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa9acde-75ca-4422-ae94-ab6dbd630313-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa9acde-75ca-4422-ae94-ab6dbd630313-istiod-ca-cert\") pod \"5aa9acde-75ca-4422-ae94-ab6dbd630313\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " Oct 30 17:11:15.184802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.184705 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-data" pod "5aa9acde-75ca-4422-ae94-ab6dbd630313" (UID: "5aa9acde-75ca-4422-ae94-ab6dbd630313"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.185143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.184850 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-socket" pod "5cf16d96-e3f2-4982-92dc-dc1188d30890" (UID: "5cf16d96-e3f2-4982-92dc-dc1188d30890"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.185143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.185060 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-data" pod "5cf16d96-e3f2-4982-92dc-dc1188d30890" (UID: "5cf16d96-e3f2-4982-92dc-dc1188d30890"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.185315 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.185285 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-certs" pod "5cf16d96-e3f2-4982-92dc-dc1188d30890" (UID: "5cf16d96-e3f2-4982-92dc-dc1188d30890"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.185608 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.185583 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf16d96-e3f2-4982-92dc-dc1188d30890-istiod-ca-cert" pod "5cf16d96-e3f2-4982-92dc-dc1188d30890" (UID: "5cf16d96-e3f2-4982-92dc-dc1188d30890"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:15.185743 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.185698 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-credential-socket" pod "5cf16d96-e3f2-4982-92dc-dc1188d30890" (UID: "5cf16d96-e3f2-4982-92dc-dc1188d30890"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.186019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.185997 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3df5215-60e4-4c09-904e-b891da8ee19f-istiod-ca-cert" pod "f3df5215-60e4-4c09-904e-b891da8ee19f" (UID: "f3df5215-60e4-4c09-904e-b891da8ee19f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:15.186132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.186081 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-certs\") pod \"f3df5215-60e4-4c09-904e-b891da8ee19f\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " Oct 30 17:11:15.186217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.186163 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-envoy\") pod \"5cf16d96-e3f2-4982-92dc-dc1188d30890\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " Oct 30 17:11:15.186217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.186204 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-credential-socket\") pod \"5aa9acde-75ca-4422-ae94-ab6dbd630313\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " Oct 30 17:11:15.186366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.186281 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-socket\") pod \"f3df5215-60e4-4c09-904e-b891da8ee19f\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " Oct 30 17:11:15.186366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.186311 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-token\" (UniqueName: \"kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-token\") pod \"5cf16d96-e3f2-4982-92dc-dc1188d30890\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " Oct 30 17:11:15.186473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.186404 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-envoy\") pod \"f3df5215-60e4-4c09-904e-b891da8ee19f\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " Oct 30 17:11:15.186473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.186445 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-socket\") pod \"5aa9acde-75ca-4422-ae94-ab6dbd630313\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " Oct 30 17:11:15.186576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.186532 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-credential-socket\") pod \"f3df5215-60e4-4c09-904e-b891da8ee19f\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " Oct 30 17:11:15.186576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.186568 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-kube-api-access-xrvnc\" (UniqueName: \"kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-kube-api-access-xrvnc\") pod \"5aa9acde-75ca-4422-ae94-ab6dbd630313\" (UID: \"5aa9acde-75ca-4422-ae94-ab6dbd630313\") " Oct 30 17:11:15.187079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.186600 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-kube-api-access-2zl4v\" (UniqueName: \"kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-kube-api-access-2zl4v\") pod \"5cf16d96-e3f2-4982-92dc-dc1188d30890\" (UID: \"5cf16d96-e3f2-4982-92dc-dc1188d30890\") " Oct 30 17:11:15.187079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.186650 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-kube-api-access-qvzfp\" (UniqueName: \"kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-kube-api-access-qvzfp\") pod \"f3df5215-60e4-4c09-904e-b891da8ee19f\" (UID: \"f3df5215-60e4-4c09-904e-b891da8ee19f\") " Oct 30 17:11:15.187576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.187556 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.187653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.187579 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.187653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.187592 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.187653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.187604 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.187653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.187618 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.187653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.187631 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.187653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.187644 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cf16d96-e3f2-4982-92dc-dc1188d30890-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.187995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.187657 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3df5215-60e4-4c09-904e-b891da8ee19f-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.191729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.191692 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-socket" pod "5aa9acde-75ca-4422-ae94-ab6dbd630313" (UID: "5aa9acde-75ca-4422-ae94-ab6dbd630313"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.191843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.191745 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa9acde-75ca-4422-ae94-ab6dbd630313-istiod-ca-cert" pod "5aa9acde-75ca-4422-ae94-ab6dbd630313" (UID: "5aa9acde-75ca-4422-ae94-ab6dbd630313"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:15.192020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.191998 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-data" pod "f3df5215-60e4-4c09-904e-b891da8ee19f" (UID: "f3df5215-60e4-4c09-904e-b891da8ee19f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.192555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.192433 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-certs" pod "f3df5215-60e4-4c09-904e-b891da8ee19f" (UID: "f3df5215-60e4-4c09-904e-b891da8ee19f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.197094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.192720 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-socket" pod "f3df5215-60e4-4c09-904e-b891da8ee19f" (UID: "f3df5215-60e4-4c09-904e-b891da8ee19f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.197094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.193019 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-credential-socket" pod "5aa9acde-75ca-4422-ae94-ab6dbd630313" (UID: "5aa9acde-75ca-4422-ae94-ab6dbd630313"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.197094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.195066 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-credential-socket" pod "f3df5215-60e4-4c09-904e-b891da8ee19f" (UID: "f3df5215-60e4-4c09-904e-b891da8ee19f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.197094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.196841 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-kube-api-access-2zl4v" pod "5cf16d96-e3f2-4982-92dc-dc1188d30890" (UID: "5cf16d96-e3f2-4982-92dc-dc1188d30890"). InnerVolumeSpecName "kube-api-access-2zl4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:15.201996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.201970 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:11:15.208800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.208683 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-token" pod "5cf16d96-e3f2-4982-92dc-dc1188d30890" (UID: "5cf16d96-e3f2-4982-92dc-dc1188d30890"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:15.208800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.208767 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-envoy" pod "5cf16d96-e3f2-4982-92dc-dc1188d30890" (UID: "5cf16d96-e3f2-4982-92dc-dc1188d30890"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.210446 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.210406 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-envoy" pod "f3df5215-60e4-4c09-904e-b891da8ee19f" (UID: "f3df5215-60e4-4c09-904e-b891da8ee19f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.212370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.212348 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-podinfo" pod "5cf16d96-e3f2-4982-92dc-dc1188d30890" (UID: "5cf16d96-e3f2-4982-92dc-dc1188d30890"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:15.212510 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.212411 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-podinfo" pod "5aa9acde-75ca-4422-ae94-ab6dbd630313" (UID: "5aa9acde-75ca-4422-ae94-ab6dbd630313"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:15.212619 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.212594 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-token" pod "5aa9acde-75ca-4422-ae94-ab6dbd630313" (UID: "5aa9acde-75ca-4422-ae94-ab6dbd630313"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:15.216035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.216008 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-kube-api-access-xrvnc" pod "5aa9acde-75ca-4422-ae94-ab6dbd630313" (UID: "5aa9acde-75ca-4422-ae94-ab6dbd630313"). InnerVolumeSpecName "kube-api-access-xrvnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:15.216153 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.216101 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-token" pod "f3df5215-60e4-4c09-904e-b891da8ee19f" (UID: "f3df5215-60e4-4c09-904e-b891da8ee19f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:15.216239 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.216213 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-podinfo" pod "f3df5215-60e4-4c09-904e-b891da8ee19f" (UID: "f3df5215-60e4-4c09-904e-b891da8ee19f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:15.216405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.216388 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-envoy" pod "5aa9acde-75ca-4422-ae94-ab6dbd630313" (UID: "5aa9acde-75ca-4422-ae94-ab6dbd630313"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.216492 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.216416 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-kube-api-access-qvzfp" pod "f3df5215-60e4-4c09-904e-b891da8ee19f" (UID: "f3df5215-60e4-4c09-904e-b891da8ee19f"). InnerVolumeSpecName "kube-api-access-qvzfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.291869 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e88601bd-82a5-47c7-be65-67915627db91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88601bd-82a5-47c7-be65-67915627db91-istiod-ca-cert\") pod \"e88601bd-82a5-47c7-be65-67915627db91\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.291998 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-data\") pod \"e88601bd-82a5-47c7-be65-67915627db91\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292017 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-istio-token\" (UniqueName: \"kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-istio-token\") pod \"e88601bd-82a5-47c7-be65-67915627db91\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292113 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-credential-socket\") pod \"e88601bd-82a5-47c7-be65-67915627db91\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292155 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-envoy\") pod \"e88601bd-82a5-47c7-be65-67915627db91\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292198 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-socket\") pod \"e88601bd-82a5-47c7-be65-67915627db91\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292231 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-kube-api-access-fh92c\" (UniqueName: \"kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-kube-api-access-fh92c\") pod \"e88601bd-82a5-47c7-be65-67915627db91\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292270 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-certs\") pod \"e88601bd-82a5-47c7-be65-67915627db91\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292331 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e88601bd-82a5-47c7-be65-67915627db91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88601bd-82a5-47c7-be65-67915627db91-istio-podinfo\") pod \"e88601bd-82a5-47c7-be65-67915627db91\" (UID: \"e88601bd-82a5-47c7-be65-67915627db91\") " Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292848 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292863 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292872 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa9acde-75ca-4422-ae94-ab6dbd630313-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292883 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292892 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292923 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292937 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292950 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292965 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.292969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292976 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa9acde-75ca-4422-ae94-ab6dbd630313-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.293881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.292990 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3df5215-60e4-4c09-904e-b891da8ee19f-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.293881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.293004 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrvnc\" (UniqueName: \"kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-kube-api-access-xrvnc\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.293881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.293016 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zl4v\" (UniqueName: \"kubernetes.io/projected/5cf16d96-e3f2-4982-92dc-dc1188d30890-kube-api-access-2zl4v\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.293881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.293030 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvzfp\" (UniqueName: \"kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-kube-api-access-qvzfp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.293881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.293042 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.293881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.293055 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.293881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.293065 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3df5215-60e4-4c09-904e-b891da8ee19f-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.293881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.293076 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa9acde-75ca-4422-ae94-ab6dbd630313-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.293881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.293088 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cf16d96-e3f2-4982-92dc-dc1188d30890-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.294324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.294243 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-credential-socket" pod "e88601bd-82a5-47c7-be65-67915627db91" (UID: "e88601bd-82a5-47c7-be65-67915627db91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.294701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.294535 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-data" pod "e88601bd-82a5-47c7-be65-67915627db91" (UID: "e88601bd-82a5-47c7-be65-67915627db91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.295056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.295027 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-socket" pod "e88601bd-82a5-47c7-be65-67915627db91" (UID: "e88601bd-82a5-47c7-be65-67915627db91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.295300 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.295101 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88601bd-82a5-47c7-be65-67915627db91-istiod-ca-cert" pod "e88601bd-82a5-47c7-be65-67915627db91" (UID: "e88601bd-82a5-47c7-be65-67915627db91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:15.295831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.295797 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-certs" pod "e88601bd-82a5-47c7-be65-67915627db91" (UID: "e88601bd-82a5-47c7-be65-67915627db91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.297655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.297550 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-envoy" pod "e88601bd-82a5-47c7-be65-67915627db91" (UID: "e88601bd-82a5-47c7-be65-67915627db91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:15.298298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.298275 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e88601bd-82a5-47c7-be65-67915627db91-istio-podinfo" pod "e88601bd-82a5-47c7-be65-67915627db91" (UID: "e88601bd-82a5-47c7-be65-67915627db91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:15.298366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.298311 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-istio-token" pod "e88601bd-82a5-47c7-be65-67915627db91" (UID: "e88601bd-82a5-47c7-be65-67915627db91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:15.300474 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.300447 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-kube-api-access-fh92c" pod "e88601bd-82a5-47c7-be65-67915627db91" (UID: "e88601bd-82a5-47c7-be65-67915627db91"). InnerVolumeSpecName "kube-api-access-fh92c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:15.395229 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.395193 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88601bd-82a5-47c7-be65-67915627db91-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.395229 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.395224 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.395229 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.395233 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.395512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.395241 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.395512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.395250 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.395512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.395258 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.395512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.395268 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh92c\" (UniqueName: \"kubernetes.io/projected/e88601bd-82a5-47c7-be65-67915627db91-kube-api-access-fh92c\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.395512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.395277 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88601bd-82a5-47c7-be65-67915627db91-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.395512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.395287 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88601bd-82a5-47c7-be65-67915627db91-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:15.443057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.443018 4828 generic.go:356] "Generic (PLEG): container finished" podID="e88601bd-82a5-47c7-be65-67915627db91" containerID="4c747700874f5ff6f4905b4afca2cfd882305632ec2cd42befdb1b3dc7da4f07" exitCode=0 Oct 30 17:11:15.443224 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.443085 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" Oct 30 17:11:15.443224 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.443162 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" event={"ID":"e88601bd-82a5-47c7-be65-67915627db91","Type":"ContainerDied","Data":"4c747700874f5ff6f4905b4afca2cfd882305632ec2cd42befdb1b3dc7da4f07"} Oct 30 17:11:15.443311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.443285 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl" event={"ID":"e88601bd-82a5-47c7-be65-67915627db91","Type":"ContainerDied","Data":"3197dbd5fcd51a72ddd2e1c8ab08b5771ddd73d8b9d24df042e035382792ef3a"} Oct 30 17:11:15.443351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.443308 4828 scope.go:122] "RemoveContainer" containerID="4c747700874f5ff6f4905b4afca2cfd882305632ec2cd42befdb1b3dc7da4f07" Oct 30 17:11:15.447039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.446777 4828 generic.go:356] "Generic (PLEG): container finished" podID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerID="3b8dc5a708fbd6075978e0d343b6ef59271d95877bd04a983496b18a0cacfab8" exitCode=0 Oct 30 17:11:15.447039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.446806 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" event={"ID":"f3df5215-60e4-4c09-904e-b891da8ee19f","Type":"ContainerDied","Data":"3b8dc5a708fbd6075978e0d343b6ef59271d95877bd04a983496b18a0cacfab8"} Oct 30 17:11:15.447039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.446842 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" Oct 30 17:11:15.447039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.446877 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk" event={"ID":"f3df5215-60e4-4c09-904e-b891da8ee19f","Type":"ContainerDied","Data":"8749e85069b7e909827a1e7f671f8e452b3fa43ea98849ef8783a75743dc9678"} Oct 30 17:11:15.449740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.449716 4828 generic.go:356] "Generic (PLEG): container finished" podID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerID="42f48b1c105aaa2a258b64c48051c8836d3005379547f24d41ff4033a2e7ee50" exitCode=0 Oct 30 17:11:15.449840 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.449804 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" event={"ID":"5cf16d96-e3f2-4982-92dc-dc1188d30890","Type":"ContainerDied","Data":"42f48b1c105aaa2a258b64c48051c8836d3005379547f24d41ff4033a2e7ee50"} Oct 30 17:11:15.449947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.449848 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" event={"ID":"5cf16d96-e3f2-4982-92dc-dc1188d30890","Type":"ContainerDied","Data":"b1bba6a327485b1860341ec553f0a3ac04533c881efa1ee59c5d1caa13c3cd1d"} Oct 30 17:11:15.450002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.449957 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t" Oct 30 17:11:15.454457 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.454437 4828 generic.go:356] "Generic (PLEG): container finished" podID="5aa9acde-75ca-4422-ae94-ab6dbd630313" containerID="64f0496dd22662f2d854bf8970100ba8b29f9f392bb15dda6de1245a0e4e2252" exitCode=0 Oct 30 17:11:15.454560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.454496 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" Oct 30 17:11:15.454560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.454502 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" event={"ID":"5aa9acde-75ca-4422-ae94-ab6dbd630313","Type":"ContainerDied","Data":"64f0496dd22662f2d854bf8970100ba8b29f9f392bb15dda6de1245a0e4e2252"} Oct 30 17:11:15.454560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.454539 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" event={"ID":"5aa9acde-75ca-4422-ae94-ab6dbd630313","Type":"ContainerDied","Data":"1ed9997b3fa69d28bf0260976b01838cd785d39ddb5dc9b1e30ccee8d57af295"} Oct 30 17:11:15.486557 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.486532 4828 scope.go:122] "RemoveContainer" containerID="4c747700874f5ff6f4905b4afca2cfd882305632ec2cd42befdb1b3dc7da4f07" Oct 30 17:11:15.487205 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:11:15.487118 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c747700874f5ff6f4905b4afca2cfd882305632ec2cd42befdb1b3dc7da4f07\": container with ID starting with 4c747700874f5ff6f4905b4afca2cfd882305632ec2cd42befdb1b3dc7da4f07 not found: ID does not exist" containerID="4c747700874f5ff6f4905b4afca2cfd882305632ec2cd42befdb1b3dc7da4f07" Oct 30 17:11:15.487429 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.487400 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c747700874f5ff6f4905b4afca2cfd882305632ec2cd42befdb1b3dc7da4f07"} err="failed to get container status \"4c747700874f5ff6f4905b4afca2cfd882305632ec2cd42befdb1b3dc7da4f07\": rpc error: code = NotFound desc = could not find container \"4c747700874f5ff6f4905b4afca2cfd882305632ec2cd42befdb1b3dc7da4f07\": container with ID starting with 4c747700874f5ff6f4905b4afca2cfd882305632ec2cd42befdb1b3dc7da4f07 not found: ID does not exist" Oct 30 17:11:15.487554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.487541 4828 scope.go:122] "RemoveContainer" containerID="3b8dc5a708fbd6075978e0d343b6ef59271d95877bd04a983496b18a0cacfab8" Oct 30 17:11:15.500102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.500078 4828 scope.go:122] "RemoveContainer" containerID="3b8dc5a708fbd6075978e0d343b6ef59271d95877bd04a983496b18a0cacfab8" Oct 30 17:11:15.504228 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:11:15.504194 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8dc5a708fbd6075978e0d343b6ef59271d95877bd04a983496b18a0cacfab8\": container with ID starting with 3b8dc5a708fbd6075978e0d343b6ef59271d95877bd04a983496b18a0cacfab8 not found: ID does not exist" containerID="3b8dc5a708fbd6075978e0d343b6ef59271d95877bd04a983496b18a0cacfab8" Oct 30 17:11:15.504359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.504239 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8dc5a708fbd6075978e0d343b6ef59271d95877bd04a983496b18a0cacfab8"} err="failed to get container status \"3b8dc5a708fbd6075978e0d343b6ef59271d95877bd04a983496b18a0cacfab8\": rpc error: code = NotFound desc = could not find container \"3b8dc5a708fbd6075978e0d343b6ef59271d95877bd04a983496b18a0cacfab8\": container with ID starting with 3b8dc5a708fbd6075978e0d343b6ef59271d95877bd04a983496b18a0cacfab8 not found: ID does not exist" Oct 30 17:11:15.504359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.504269 4828 scope.go:122] "RemoveContainer" containerID="42f48b1c105aaa2a258b64c48051c8836d3005379547f24d41ff4033a2e7ee50" Oct 30 17:11:15.507988 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:11:15.507944 4828 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa9acde_75ca_4422_ae94_ab6dbd630313.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf16d96_e3f2_4982_92dc_dc1188d30890.slice\": RecentStats: unable to find data in memory cache]" Oct 30 17:11:15.519782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.519742 4828 scope.go:122] "RemoveContainer" containerID="42f48b1c105aaa2a258b64c48051c8836d3005379547f24d41ff4033a2e7ee50" Oct 30 17:11:15.520089 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:11:15.520069 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f48b1c105aaa2a258b64c48051c8836d3005379547f24d41ff4033a2e7ee50\": container with ID starting with 42f48b1c105aaa2a258b64c48051c8836d3005379547f24d41ff4033a2e7ee50 not found: ID does not exist" containerID="42f48b1c105aaa2a258b64c48051c8836d3005379547f24d41ff4033a2e7ee50" Oct 30 17:11:15.520193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.520097 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f48b1c105aaa2a258b64c48051c8836d3005379547f24d41ff4033a2e7ee50"} err="failed to get container status \"42f48b1c105aaa2a258b64c48051c8836d3005379547f24d41ff4033a2e7ee50\": rpc error: code = NotFound desc = could not find container \"42f48b1c105aaa2a258b64c48051c8836d3005379547f24d41ff4033a2e7ee50\": container with ID starting with 42f48b1c105aaa2a258b64c48051c8836d3005379547f24d41ff4033a2e7ee50 not found: ID does not exist" Oct 30 17:11:15.520193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.520114 4828 scope.go:122] "RemoveContainer" containerID="64f0496dd22662f2d854bf8970100ba8b29f9f392bb15dda6de1245a0e4e2252" Oct 30 17:11:15.534135 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.534105 4828 scope.go:122] "RemoveContainer" containerID="64f0496dd22662f2d854bf8970100ba8b29f9f392bb15dda6de1245a0e4e2252" Oct 30 17:11:15.535351 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:11:15.535319 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f0496dd22662f2d854bf8970100ba8b29f9f392bb15dda6de1245a0e4e2252\": container with ID starting with 64f0496dd22662f2d854bf8970100ba8b29f9f392bb15dda6de1245a0e4e2252 not found: ID does not exist" containerID="64f0496dd22662f2d854bf8970100ba8b29f9f392bb15dda6de1245a0e4e2252" Oct 30 17:11:15.535471 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.535361 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f0496dd22662f2d854bf8970100ba8b29f9f392bb15dda6de1245a0e4e2252"} err="failed to get container status \"64f0496dd22662f2d854bf8970100ba8b29f9f392bb15dda6de1245a0e4e2252\": rpc error: code = NotFound desc = could not find container \"64f0496dd22662f2d854bf8970100ba8b29f9f392bb15dda6de1245a0e4e2252\": container with ID starting with 64f0496dd22662f2d854bf8970100ba8b29f9f392bb15dda6de1245a0e4e2252 not found: ID does not exist" Oct 30 17:11:15.552554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.552467 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl"] Oct 30 17:11:15.558198 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.558170 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-4ghfl"] Oct 30 17:11:15.592779 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.592743 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t"] Oct 30 17:11:15.598450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.598419 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-jjv2t"] Oct 30 17:11:15.632148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.632117 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk"] Oct 30 17:11:15.637305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.637276 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-qfbzk"] Oct 30 17:11:15.673187 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.673149 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d"] Oct 30 17:11:15.679825 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.679783 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d"] Oct 30 17:11:15.959825 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.959738 4828 patch_prober.go:29] interesting pod/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:11:15.959825 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:15.959802 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-86f50a-openshift-default-67c6695475-cds7d" podUID="5aa9acde-75ca-4422-ae94-ab6dbd630313" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:16.286951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:16.286825 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:16.494864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:16.494825 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa9acde-75ca-4422-ae94-ab6dbd630313" path="/var/lib/kubelet/pods/5aa9acde-75ca-4422-ae94-ab6dbd630313/volumes" Oct 30 17:11:16.495654 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:16.495631 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" path="/var/lib/kubelet/pods/5cf16d96-e3f2-4982-92dc-dc1188d30890/volumes" Oct 30 17:11:16.496342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:16.496324 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e88601bd-82a5-47c7-be65-67915627db91" path="/var/lib/kubelet/pods/e88601bd-82a5-47c7-be65-67915627db91/volumes" Oct 30 17:11:16.497377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:16.497357 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" path="/var/lib/kubelet/pods/f3df5215-60e4-4c09-904e-b891da8ee19f/volumes" Oct 30 17:11:18.321732 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:18.319428 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:18.934716 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:18.934682 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:21.365254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:21.365213 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:22.402645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:22.402596 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r26lb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:11:22.403307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:22.403268 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" podUID="1fc9a9e7-416b-4b87-912e-a524f41e0d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:22.403877 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:22.403847 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:11:22.835136 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:22.807803 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:23.872490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:23.872451 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:23.873787 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:23.873752 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c"] Oct 30 17:11:23.874317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:23.874292 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" containerID="cri-o://f6e343f433043cdd5c45db77d5a482f88b2c6335b55db60a8d8e30f309248897" gracePeriod=30 Oct 30 17:11:24.067069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:24.067035 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn"] Oct 30 17:11:24.067706 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:24.067654 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" containerID="cri-o://cde8a492a4e059669a42c8d5417cd32e28347e7208d17d1e10892e780b98b797" gracePeriod=30 Oct 30 17:11:24.114735 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:24.114696 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh"] Oct 30 17:11:24.115351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:24.115290 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" podUID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" containerName="istio-proxy" containerID="cri-o://e1b53e79bc8573079b8f86046e1c89115521ea836d97344cce1560e6de85f76c" gracePeriod=30 Oct 30 17:11:24.132130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:24.131592 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg"] Oct 30 17:11:24.132130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:24.131845 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" podUID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" containerName="istio-proxy" containerID="cri-o://559d68d9af1d6223213120b8083ff41c517d9f2f349d5d7f995534f834749ea8" gracePeriod=30 Oct 30 17:11:24.467508 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:24.467392 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" Oct 30 17:11:25.691637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:25.691584 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:26.720548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:26.720514 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:26.734481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:26.734446 4828 patch_prober.go:29] interesting pod/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:11:26.734662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:26.734559 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:11:27.521142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:27.521038 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:29.233634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.233527 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:29.269302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.268500 4828 generic.go:356] "Generic (PLEG): container finished" podID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerID="f6e343f433043cdd5c45db77d5a482f88b2c6335b55db60a8d8e30f309248897" exitCode=0 Oct 30 17:11:29.269302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.268640 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" event={"ID":"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542","Type":"ContainerDied","Data":"f6e343f433043cdd5c45db77d5a482f88b2c6335b55db60a8d8e30f309248897"} Oct 30 17:11:29.271567 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.271207 4828 generic.go:356] "Generic (PLEG): container finished" podID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerID="cde8a492a4e059669a42c8d5417cd32e28347e7208d17d1e10892e780b98b797" exitCode=0 Oct 30 17:11:29.271567 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.271488 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" event={"ID":"42886d0b-4093-4f40-a8d3-9fc61d25daab","Type":"ContainerDied","Data":"cde8a492a4e059669a42c8d5417cd32e28347e7208d17d1e10892e780b98b797"} Oct 30 17:11:29.505966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.503702 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:11:29.633682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.633651 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:11:29.641447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.641417 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-certs\") pod \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " Oct 30 17:11:29.641554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.641466 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-podinfo\") pod \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " Oct 30 17:11:29.641617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.641592 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-envoy\") pod \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " Oct 30 17:11:29.641669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.641634 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-kube-api-access-mc85k\" (UniqueName: \"kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-kube-api-access-mc85k\") pod \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " Oct 30 17:11:29.641805 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.641781 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-socket\") pod \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " Oct 30 17:11:29.641866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.641788 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-certs" pod "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" (UID: "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.641997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.641893 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-data\") pod \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " Oct 30 17:11:29.641997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.641973 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-token\" (UniqueName: \"kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-token\") pod \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " Oct 30 17:11:29.642483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.642408 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-socket" pod "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" (UID: "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.643284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.642748 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-credential-socket\") pod \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " Oct 30 17:11:29.643284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.642803 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istiod-ca-cert\") pod \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\" (UID: \"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542\") " Oct 30 17:11:29.644376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.643892 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.644376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.643941 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.644376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.642800 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-data" pod "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" (UID: "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.644851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.644666 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-credential-socket" pod "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" (UID: "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.645443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.645090 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istiod-ca-cert" pod "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" (UID: "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:29.646345 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.646311 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-kube-api-access-mc85k" pod "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" (UID: "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542"). InnerVolumeSpecName "kube-api-access-mc85k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:29.660265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.654074 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-podinfo" pod "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" (UID: "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:29.660265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.659312 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-token" pod "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" (UID: "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:29.663672 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.663605 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-envoy" pod "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" (UID: "6f912ad4-9ec4-4c7d-bd49-2d81e2e99542"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.745162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.744735 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-credential-socket\") pod \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " Oct 30 17:11:29.745162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.744792 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-podinfo\") pod \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " Oct 30 17:11:29.745162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.744874 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-envoy\") pod \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " Oct 30 17:11:29.745162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.744945 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-socket\") pod \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " Oct 30 17:11:29.745162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.745170 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istiod-ca-cert\") pod \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " Oct 30 17:11:29.745568 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.745217 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-data\") pod \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " Oct 30 17:11:29.745568 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.745295 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-token\" (UniqueName: \"kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-token\") pod \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " Oct 30 17:11:29.745568 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.745350 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-kube-api-access-fbjkt\" (UniqueName: \"kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-kube-api-access-fbjkt\") pod \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " Oct 30 17:11:29.745568 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.745470 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-certs\") pod \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\" (UID: \"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917\") " Oct 30 17:11:29.750795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.746283 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.750795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.746312 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.750795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.746326 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc85k\" (UniqueName: \"kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-kube-api-access-mc85k\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.750795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.746341 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.750795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.746353 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.750795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.746366 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.750795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.746382 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.750795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.746833 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-socket" pod "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" (UID: "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.750795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.747053 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-credential-socket" pod "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" (UID: "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.750795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.747390 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-certs" pod "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" (UID: "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.750795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.747721 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istiod-ca-cert" pod "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" (UID: "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:29.750795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.747983 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-data" pod "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" (UID: "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.760092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.759913 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-kube-api-access-fbjkt" pod "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" (UID: "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917"). InnerVolumeSpecName "kube-api-access-fbjkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:29.760962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.760764 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-envoy" pod "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" (UID: "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:29.760962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.760873 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-podinfo" pod "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" (UID: "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:29.761697 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.761665 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-token" pod "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" (UID: "a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:29.848831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.848660 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.848831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.848703 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.848831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.848716 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.848831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.848730 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbjkt\" (UniqueName: \"kubernetes.io/projected/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-kube-api-access-fbjkt\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.848831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.848746 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.848831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.848760 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.848831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.848773 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.848831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.848786 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.848831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.848802 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:29.958015 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.957978 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:11:29.988633 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:29.988599 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:11:30.052043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.052000 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-kube-api-access-c7cnn\" (UniqueName: \"kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-kube-api-access-c7cnn\") pod \"42886d0b-4093-4f40-a8d3-9fc61d25daab\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " Oct 30 17:11:30.052213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.052083 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-token\" (UniqueName: \"kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-token\") pod \"42886d0b-4093-4f40-a8d3-9fc61d25daab\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " Oct 30 17:11:30.052643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.052617 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-kube-api-access-scsm4\" (UniqueName: \"kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-kube-api-access-scsm4\") pod \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " Oct 30 17:11:30.052728 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.052704 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-certs\") pod \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " Oct 30 17:11:30.052784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.052751 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-podinfo\") pod \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " Oct 30 17:11:30.052839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.052808 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-envoy\") pod \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " Oct 30 17:11:30.052948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.052863 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-data\") pod \"42886d0b-4093-4f40-a8d3-9fc61d25daab\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " Oct 30 17:11:30.052948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.052912 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-socket\") pod \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " Oct 30 17:11:30.053103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.053081 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-socket\") pod \"42886d0b-4093-4f40-a8d3-9fc61d25daab\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " Oct 30 17:11:30.053226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.053130 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-credential-socket\") pod \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " Oct 30 17:11:30.053226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.053167 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-certs\") pod \"42886d0b-4093-4f40-a8d3-9fc61d25daab\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " Oct 30 17:11:30.053335 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.053222 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-data\") pod \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " Oct 30 17:11:30.053335 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.053294 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-envoy\") pod \"42886d0b-4093-4f40-a8d3-9fc61d25daab\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " Oct 30 17:11:30.053443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.053412 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-token\" (UniqueName: \"kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-token\") pod \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " Oct 30 17:11:30.053497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.053447 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-podinfo\") pod \"42886d0b-4093-4f40-a8d3-9fc61d25daab\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " Oct 30 17:11:30.053720 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.053692 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42886d0b-4093-4f40-a8d3-9fc61d25daab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42886d0b-4093-4f40-a8d3-9fc61d25daab-istiod-ca-cert\") pod \"42886d0b-4093-4f40-a8d3-9fc61d25daab\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " Oct 30 17:11:30.053798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.053782 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-credential-socket\") pod \"42886d0b-4093-4f40-a8d3-9fc61d25daab\" (UID: \"42886d0b-4093-4f40-a8d3-9fc61d25daab\") " Oct 30 17:11:30.053862 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.053789 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-socket" pod "3108a244-2537-47cb-a2dd-01b73c5cd3e9" (UID: "3108a244-2537-47cb-a2dd-01b73c5cd3e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:30.053862 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.053824 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istiod-ca-cert\") pod \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\" (UID: \"3108a244-2537-47cb-a2dd-01b73c5cd3e9\") " Oct 30 17:11:30.054092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.054052 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-certs" pod "3108a244-2537-47cb-a2dd-01b73c5cd3e9" (UID: "3108a244-2537-47cb-a2dd-01b73c5cd3e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:30.054459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.054437 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istiod-ca-cert" pod "3108a244-2537-47cb-a2dd-01b73c5cd3e9" (UID: "3108a244-2537-47cb-a2dd-01b73c5cd3e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:30.055087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.054999 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.055087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.055028 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.055087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.055042 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.055566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.055419 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-data" pod "42886d0b-4093-4f40-a8d3-9fc61d25daab" (UID: "42886d0b-4093-4f40-a8d3-9fc61d25daab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:30.055650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.055592 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-socket" pod "42886d0b-4093-4f40-a8d3-9fc61d25daab" (UID: "42886d0b-4093-4f40-a8d3-9fc61d25daab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:30.056003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.055806 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-kube-api-access-scsm4" pod "3108a244-2537-47cb-a2dd-01b73c5cd3e9" (UID: "3108a244-2537-47cb-a2dd-01b73c5cd3e9"). InnerVolumeSpecName "kube-api-access-scsm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:30.056151 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.056098 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-envoy" pod "3108a244-2537-47cb-a2dd-01b73c5cd3e9" (UID: "3108a244-2537-47cb-a2dd-01b73c5cd3e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:30.056351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.056272 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-certs" pod "42886d0b-4093-4f40-a8d3-9fc61d25daab" (UID: "42886d0b-4093-4f40-a8d3-9fc61d25daab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:30.056351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.056347 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-data" pod "3108a244-2537-47cb-a2dd-01b73c5cd3e9" (UID: "3108a244-2537-47cb-a2dd-01b73c5cd3e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:30.056769 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.056550 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-credential-socket" pod "3108a244-2537-47cb-a2dd-01b73c5cd3e9" (UID: "3108a244-2537-47cb-a2dd-01b73c5cd3e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:30.056769 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.056594 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-podinfo" pod "3108a244-2537-47cb-a2dd-01b73c5cd3e9" (UID: "3108a244-2537-47cb-a2dd-01b73c5cd3e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:30.056769 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.056622 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42886d0b-4093-4f40-a8d3-9fc61d25daab-istiod-ca-cert" pod "42886d0b-4093-4f40-a8d3-9fc61d25daab" (UID: "42886d0b-4093-4f40-a8d3-9fc61d25daab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:30.057127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.057092 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-credential-socket" pod "42886d0b-4093-4f40-a8d3-9fc61d25daab" (UID: "42886d0b-4093-4f40-a8d3-9fc61d25daab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:30.058232 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.058208 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-token" pod "42886d0b-4093-4f40-a8d3-9fc61d25daab" (UID: "42886d0b-4093-4f40-a8d3-9fc61d25daab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:30.058378 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.058356 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-token" pod "3108a244-2537-47cb-a2dd-01b73c5cd3e9" (UID: "3108a244-2537-47cb-a2dd-01b73c5cd3e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:30.058660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.058634 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-envoy" pod "42886d0b-4093-4f40-a8d3-9fc61d25daab" (UID: "42886d0b-4093-4f40-a8d3-9fc61d25daab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:30.059201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.059166 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-podinfo" pod "42886d0b-4093-4f40-a8d3-9fc61d25daab" (UID: "42886d0b-4093-4f40-a8d3-9fc61d25daab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:30.060189 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.060168 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-kube-api-access-c7cnn" pod "42886d0b-4093-4f40-a8d3-9fc61d25daab" (UID: "42886d0b-4093-4f40-a8d3-9fc61d25daab"). InnerVolumeSpecName "kube-api-access-c7cnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:30.156409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156373 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42886d0b-4093-4f40-a8d3-9fc61d25daab-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156406 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156420 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7cnn\" (UniqueName: \"kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-kube-api-access-c7cnn\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156434 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156447 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scsm4\" (UniqueName: \"kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-kube-api-access-scsm4\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156458 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156468 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156480 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156494 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156507 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156519 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156530 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156541 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156551 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3108a244-2537-47cb-a2dd-01b73c5cd3e9-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.156638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.156564 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42886d0b-4093-4f40-a8d3-9fc61d25daab-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:11:30.284280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.284244 4828 generic.go:356] "Generic (PLEG): container finished" podID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" containerID="559d68d9af1d6223213120b8083ff41c517d9f2f349d5d7f995534f834749ea8" exitCode=0 Oct 30 17:11:30.284764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.284300 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" Oct 30 17:11:30.284764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.284339 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" event={"ID":"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917","Type":"ContainerDied","Data":"559d68d9af1d6223213120b8083ff41c517d9f2f349d5d7f995534f834749ea8"} Oct 30 17:11:30.284764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.284415 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg" event={"ID":"a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917","Type":"ContainerDied","Data":"07d900adf172fd55f71771662dcb6c65899d5af2c347f1f46075b2e775a0fe47"} Oct 30 17:11:30.284764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.284462 4828 scope.go:122] "RemoveContainer" containerID="559d68d9af1d6223213120b8083ff41c517d9f2f349d5d7f995534f834749ea8" Oct 30 17:11:30.286742 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.286711 4828 generic.go:356] "Generic (PLEG): container finished" podID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" containerID="e1b53e79bc8573079b8f86046e1c89115521ea836d97344cce1560e6de85f76c" exitCode=0 Oct 30 17:11:30.286865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.286826 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" Oct 30 17:11:30.287079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.287053 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" event={"ID":"3108a244-2537-47cb-a2dd-01b73c5cd3e9","Type":"ContainerDied","Data":"e1b53e79bc8573079b8f86046e1c89115521ea836d97344cce1560e6de85f76c"} Oct 30 17:11:30.287166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.287113 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh" event={"ID":"3108a244-2537-47cb-a2dd-01b73c5cd3e9","Type":"ContainerDied","Data":"b26e083c56d8a013bf71b15ab173ba936dd810587b72f89bc68ffe881e3d97c2"} Oct 30 17:11:30.296057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.289048 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" event={"ID":"6f912ad4-9ec4-4c7d-bd49-2d81e2e99542","Type":"ContainerDied","Data":"8d12196d6e16f0da6bbb7d4a52d4cb66414ca7a3cc82c3804c771760ef9459ac"} Oct 30 17:11:30.296057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.289159 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c" Oct 30 17:11:30.297547 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.297514 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" event={"ID":"42886d0b-4093-4f40-a8d3-9fc61d25daab","Type":"ContainerDied","Data":"e486477b590cf24cc248f859bd2653a018f373e4f91db4af2d8ff2b9d59f189c"} Oct 30 17:11:30.297662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.297622 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn" Oct 30 17:11:30.304339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.303275 4828 scope.go:122] "RemoveContainer" containerID="559d68d9af1d6223213120b8083ff41c517d9f2f349d5d7f995534f834749ea8" Oct 30 17:11:30.304339 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:11:30.303650 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559d68d9af1d6223213120b8083ff41c517d9f2f349d5d7f995534f834749ea8\": container with ID starting with 559d68d9af1d6223213120b8083ff41c517d9f2f349d5d7f995534f834749ea8 not found: ID does not exist" containerID="559d68d9af1d6223213120b8083ff41c517d9f2f349d5d7f995534f834749ea8" Oct 30 17:11:30.304339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.303686 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559d68d9af1d6223213120b8083ff41c517d9f2f349d5d7f995534f834749ea8"} err="failed to get container status \"559d68d9af1d6223213120b8083ff41c517d9f2f349d5d7f995534f834749ea8\": rpc error: code = NotFound desc = could not find container \"559d68d9af1d6223213120b8083ff41c517d9f2f349d5d7f995534f834749ea8\": container with ID starting with 559d68d9af1d6223213120b8083ff41c517d9f2f349d5d7f995534f834749ea8 not found: ID does not exist" Oct 30 17:11:30.304339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.303709 4828 scope.go:122] "RemoveContainer" containerID="e1b53e79bc8573079b8f86046e1c89115521ea836d97344cce1560e6de85f76c" Oct 30 17:11:30.320114 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.320085 4828 scope.go:122] "RemoveContainer" containerID="e1b53e79bc8573079b8f86046e1c89115521ea836d97344cce1560e6de85f76c" Oct 30 17:11:30.320476 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:11:30.320448 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b53e79bc8573079b8f86046e1c89115521ea836d97344cce1560e6de85f76c\": container with ID starting with e1b53e79bc8573079b8f86046e1c89115521ea836d97344cce1560e6de85f76c not found: ID does not exist" containerID="e1b53e79bc8573079b8f86046e1c89115521ea836d97344cce1560e6de85f76c" Oct 30 17:11:30.320534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.320495 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b53e79bc8573079b8f86046e1c89115521ea836d97344cce1560e6de85f76c"} err="failed to get container status \"e1b53e79bc8573079b8f86046e1c89115521ea836d97344cce1560e6de85f76c\": rpc error: code = NotFound desc = could not find container \"e1b53e79bc8573079b8f86046e1c89115521ea836d97344cce1560e6de85f76c\": container with ID starting with e1b53e79bc8573079b8f86046e1c89115521ea836d97344cce1560e6de85f76c not found: ID does not exist" Oct 30 17:11:30.320534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.320522 4828 scope.go:122] "RemoveContainer" containerID="f6e343f433043cdd5c45db77d5a482f88b2c6335b55db60a8d8e30f309248897" Oct 30 17:11:30.331659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.331636 4828 scope.go:122] "RemoveContainer" containerID="cde8a492a4e059669a42c8d5417cd32e28347e7208d17d1e10892e780b98b797" Oct 30 17:11:30.357402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.357373 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg"] Oct 30 17:11:30.369451 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.369413 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-jpqhg"] Oct 30 17:11:30.431589 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.431553 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c"] Oct 30 17:11:30.436571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.436542 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-gnp9c"] Oct 30 17:11:30.468778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.468744 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh"] Oct 30 17:11:30.474359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.474326 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qxcqh"] Oct 30 17:11:30.497591 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.497552 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" path="/var/lib/kubelet/pods/3108a244-2537-47cb-a2dd-01b73c5cd3e9/volumes" Oct 30 17:11:30.498462 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.498441 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" path="/var/lib/kubelet/pods/6f912ad4-9ec4-4c7d-bd49-2d81e2e99542/volumes" Oct 30 17:11:30.499261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.499238 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" path="/var/lib/kubelet/pods/a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917/volumes" Oct 30 17:11:30.512327 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.512300 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn"] Oct 30 17:11:30.520671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.520636 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-qcxgn"] Oct 30 17:11:30.804743 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.804369 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:30.953122 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:30.953025 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:31.312823 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:31.312789 4828 generic.go:356] "Generic (PLEG): container finished" podID="068993d6-1de9-4224-b028-39ce4fb563ea" containerID="aef9d048db08610705874f6def9ab3b811be2246deb4ac2d7995304da7b878a9" exitCode=0 Oct 30 17:11:31.313262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:31.312855 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" event={"ID":"068993d6-1de9-4224-b028-39ce4fb563ea","Type":"ContainerDied","Data":"aef9d048db08610705874f6def9ab3b811be2246deb4ac2d7995304da7b878a9"} Oct 30 17:11:31.313389 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:31.313373 4828 scope.go:122] "RemoveContainer" containerID="aef9d048db08610705874f6def9ab3b811be2246deb4ac2d7995304da7b878a9" Oct 30 17:11:31.315320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:31.315279 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:31.315449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:31.315362 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:31.315449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:31.315413 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:32.033454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:32.033415 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:11:32.324336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:32.324257 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" event={"ID":"068993d6-1de9-4224-b028-39ce4fb563ea","Type":"ContainerStarted","Data":"b81fc983f0dbd3c9512e35fd29f893e7ca544198c40c3259973c373bbbb294aa"} Oct 30 17:11:32.511112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:32.511064 4828 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 17:11:32.511432 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:32.511396 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 17:11:32.512524 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:32.512495 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" path="/var/lib/kubelet/pods/42886d0b-4093-4f40-a8d3-9fc61d25daab/volumes" Oct 30 17:11:32.513241 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:32.513220 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:11:33.013929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:33.013402 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:33.498976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:33.493305 4828 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 17:11:33.498976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:33.493382 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 17:11:34.498926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:34.493524 4828 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 17:11:34.498926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:34.493586 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 17:11:35.154921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:35.153561 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:35.494430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:35.494333 4828 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 17:11:35.494724 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:35.494695 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 17:11:36.499040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:36.499000 4828 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 17:11:36.499647 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:36.499074 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 17:11:37.099062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:37.098931 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:37.500109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:37.500014 4828 patch_prober.go:29] interesting pod/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 17:11:37.500763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:37.500728 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 17:11:37.942469 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:11:37.942424 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:11:38.829923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:38.829868 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:38.858251 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:38.858219 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:11:38.860196 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:38.860172 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:11:38.880534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:38.880501 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:11:40.517924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.514310 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.539276 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8"] Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540524 4828 cpu_manager.go:414] "containerMap: removing container" podUID="5aa9acde-75ca-4422-ae94-ab6dbd630313" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540546 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa9acde-75ca-4422-ae94-ab6dbd630313" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540563 4828 cpu_manager.go:414] "containerMap: removing container" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540573 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540602 4828 cpu_manager.go:414] "containerMap: removing container" podUID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540611 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540622 4828 cpu_manager.go:414] "containerMap: removing container" podUID="e88601bd-82a5-47c7-be65-67915627db91" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540629 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88601bd-82a5-47c7-be65-67915627db91" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540679 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540688 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540707 4828 cpu_manager.go:414] "containerMap: removing container" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540714 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540741 4828 cpu_manager.go:414] "containerMap: removing container" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540748 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540766 4828 cpu_manager.go:414] "containerMap: removing container" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540774 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540957 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3df5215-60e4-4c09-904e-b891da8ee19f" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540974 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cf16d96-e3f2-4982-92dc-dc1188d30890" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.540988 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa9acde-75ca-4422-ae94-ab6dbd630313" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.541000 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="42886d0b-4093-4f40-a8d3-9fc61d25daab" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.541013 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="3108a244-2537-47cb-a2dd-01b73c5cd3e9" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.541025 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="e88601bd-82a5-47c7-be65-67915627db91" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.541043 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a5a6a7-8d69-4f16-ba3f-8aebd74dd917" containerName="istio-proxy" Oct 30 17:11:40.541764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.541054 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f912ad4-9ec4-4c7d-bd49-2d81e2e99542" containerName="istio-proxy" Oct 30 17:11:40.545838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.545802 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8"] Oct 30 17:11:40.546053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.545891 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:40.561540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.561512 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7ebeb6e1-serving-cert\"" Oct 30 17:11:40.638602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.638563 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjxzk\" (UniqueName: \"kubernetes.io/projected/2760fc54-3612-4d2f-9e8d-c3054da972a9-kube-api-access-pjxzk\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:40.638785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.638719 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-maas-api-tls\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:40.638856 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.638837 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-metrics-certs\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:40.638955 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.638940 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2760fc54-3612-4d2f-9e8d-c3054da972a9-openshift-service-ca\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:40.703023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.702991 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:40.755014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.746351 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjxzk\" (UniqueName: \"kubernetes.io/projected/2760fc54-3612-4d2f-9e8d-c3054da972a9-kube-api-access-pjxzk\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:40.755014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.746494 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-maas-api-tls\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:40.755014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.746631 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-metrics-certs\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:40.755014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.746698 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2760fc54-3612-4d2f-9e8d-c3054da972a9-openshift-service-ca\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:40.755014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.747805 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2760fc54-3612-4d2f-9e8d-c3054da972a9-openshift-service-ca\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:40.764976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.758080 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-maas-api-tls\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:40.764976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.758699 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-metrics-certs\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:40.807968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.802314 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjxzk\" (UniqueName: \"kubernetes.io/projected/2760fc54-3612-4d2f-9e8d-c3054da972a9-kube-api-access-pjxzk\") pod \"maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:40.868584 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.868540 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:40.872490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:40.872464 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:41.190581 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:11:41.190546 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2760fc54_3612_4d2f_9e8d_c3054da972a9.slice/crio-66e3d9cb9e27d8d2c162995bdea3d45654956caf78df5403d0a4438c8075bca5 WatchSource:0}: Error finding container 66e3d9cb9e27d8d2c162995bdea3d45654956caf78df5403d0a4438c8075bca5: Status 404 returned error can't find the container with id 66e3d9cb9e27d8d2c162995bdea3d45654956caf78df5403d0a4438c8075bca5 Oct 30 17:11:41.193029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:41.192981 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8"] Oct 30 17:11:41.245754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:41.245715 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:41.408234 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:41.408199 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" event={"ID":"2760fc54-3612-4d2f-9e8d-c3054da972a9","Type":"ContainerStarted","Data":"66e3d9cb9e27d8d2c162995bdea3d45654956caf78df5403d0a4438c8075bca5"} Oct 30 17:11:41.753181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:41.753144 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:42.143960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.143918 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2"] Oct 30 17:11:42.149778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.149743 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.179659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.175098 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-db8417-openshift-default-dockercfg-94mls\"" Oct 30 17:11:42.189962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.188791 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2"] Oct 30 17:11:42.353926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.304208 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:42.362039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.357919 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-credential-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.362039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.358026 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-istio-token\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.362039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.358094 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.362039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.358186 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f119ba5-b435-453d-809e-bb2795e3066d-istiod-ca-cert\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.362039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.358256 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-certs\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.362039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.358325 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f119ba5-b435-453d-809e-bb2795e3066d-istio-podinfo\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.362039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.358441 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-envoy\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.362039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.358508 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kdln\" (UniqueName: \"kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-kube-api-access-4kdln\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.362039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.358577 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-data\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.472977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.471800 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f119ba5-b435-453d-809e-bb2795e3066d-istio-podinfo\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.472977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.471938 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-envoy\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.472977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.471988 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kdln\" (UniqueName: \"kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-kube-api-access-4kdln\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.472977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.472040 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-data\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.472977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.472123 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-credential-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.472977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.472179 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-istio-token\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.472977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.472225 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.472977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.472296 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f119ba5-b435-453d-809e-bb2795e3066d-istiod-ca-cert\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.472977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.472355 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-certs\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.472977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.472882 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-certs\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.494926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.494838 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-data\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.507965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.497082 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-credential-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.507965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.497349 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-socket\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.507965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.498657 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f119ba5-b435-453d-809e-bb2795e3066d-istiod-ca-cert\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.507965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.502802 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:42.608923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.597595 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f119ba5-b435-453d-809e-bb2795e3066d-istio-podinfo\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.608923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.599266 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-envoy\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.608923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.608476 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-istio-token\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.619958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.610128 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kdln\" (UniqueName: \"kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-kube-api-access-4kdln\") pod \"e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:42.645958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.644063 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" event={"ID":"2760fc54-3612-4d2f-9e8d-c3054da972a9","Type":"ContainerStarted","Data":"517e58b10da5766f2b081e67ad9f9ea1da63cd85a25843bfcdced614f81403e2"} Oct 30 17:11:42.646152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.645986 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:42.804475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:42.802963 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:43.077988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:43.075518 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:43.190153 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:43.189333 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" podStartSLOduration=2.789528962 podStartE2EDuration="3.189314744s" totalImagesPullingTime="399.785782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:42.69539289 +0000 UTC m=+1480.340903071" watchObservedRunningTime="2026-10-30 17:11:43.189314744 +0000 UTC m=+1480.834824925" Oct 30 17:11:43.190153 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:11:43.189691 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f119ba5_b435_453d_809e_bb2795e3066d.slice/crio-e297c1ffb88504af7167eae8f6d7d5baa7f2d7ea6c16dd57d17a09d14d9f6aa0 WatchSource:0}: Error finding container e297c1ffb88504af7167eae8f6d7d5baa7f2d7ea6c16dd57d17a09d14d9f6aa0: Status 404 returned error can't find the container with id e297c1ffb88504af7167eae8f6d7d5baa7f2d7ea6c16dd57d17a09d14d9f6aa0 Oct 30 17:11:43.190754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:43.190697 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2"] Oct 30 17:11:43.199597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:43.193206 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:43.199946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:43.199887 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:43.200140 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:43.200118 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:43.633620 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:43.633584 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" event={"ID":"6f119ba5-b435-453d-809e-bb2795e3066d","Type":"ContainerStarted","Data":"e4cc0b0e5c1b171b36712679a8469dbc926573876fdf65a825863e48f106f20f"} Oct 30 17:11:43.633832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:43.633641 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" event={"ID":"6f119ba5-b435-453d-809e-bb2795e3066d","Type":"ContainerStarted","Data":"e297c1ffb88504af7167eae8f6d7d5baa7f2d7ea6c16dd57d17a09d14d9f6aa0"} Oct 30 17:11:43.682956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:43.682855 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" podStartSLOduration=1.682836862 podStartE2EDuration="1.682836862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:43.679841682 +0000 UTC m=+1481.325351862" watchObservedRunningTime="2026-10-30 17:11:43.682836862 +0000 UTC m=+1481.328347044" Oct 30 17:11:43.808103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:43.808014 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:44.875919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:44.870088 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:45.412932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:45.409721 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:45.679720 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:45.679634 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:45.680727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:45.680704 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:11:46.590948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:46.588640 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:46.736652 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:46.736608 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:47.366921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:47.366530 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:48.052475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:48.052435 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:48.509443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:48.509408 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:48.676141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:48.676105 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:49.676291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:49.676260 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:11:49.713310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:49.713274 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:50.415185 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.415123 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22"] Oct 30 17:11:50.431926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.426061 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.504769 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.504732 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-728ad7\"/\"openshift-service-ca.crt\"" Oct 30 17:11:50.504984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.504893 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-728ad7\"/\"test-model-728ad7-kserve-self-signed-certs\"" Oct 30 17:11:50.505058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.504984 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-728ad7\"/\"kube-root-ca.crt\"" Oct 30 17:11:50.505129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.505111 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-728ad7\"/\"default-dockercfg-l8cks\"" Oct 30 17:11:50.526306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.525704 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:50.529821 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.529779 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22"] Oct 30 17:11:50.579195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.579157 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-dshm\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.579417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.579218 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-home\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.579417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.579400 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c9r4\" (UniqueName: \"kubernetes.io/projected/9875cf06-8438-4b8e-96b2-592d1ea387bf-kube-api-access-5c9r4\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.579630 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.579608 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9875cf06-8438-4b8e-96b2-592d1ea387bf-tls-certs\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.579783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.579728 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-tmp-dir\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.579875 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.579813 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-model-cache\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.688002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.684094 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-dshm\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.688002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.684151 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-home\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.688002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.684287 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c9r4\" (UniqueName: \"kubernetes.io/projected/9875cf06-8438-4b8e-96b2-592d1ea387bf-kube-api-access-5c9r4\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.688002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.684470 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9875cf06-8438-4b8e-96b2-592d1ea387bf-tls-certs\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.688002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.684554 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-tmp-dir\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.688002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.684611 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-model-cache\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.703925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.695631 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-dshm\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.703925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.695948 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-home\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.708685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.708650 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-tmp-dir\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.712287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.709795 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-model-cache\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.720932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.719855 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9875cf06-8438-4b8e-96b2-592d1ea387bf-tls-certs\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.767577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.767545 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c9r4\" (UniqueName: \"kubernetes.io/projected/9875cf06-8438-4b8e-96b2-592d1ea387bf-kube-api-access-5c9r4\") pod \"test-model-728ad7-kserve-7bf595ffb5-lkf22\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:50.844717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:50.844678 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:51.200262 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:11:51.200231 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9875cf06_8438_4b8e_96b2_592d1ea387bf.slice/crio-edff7fd72bac38fdaa2924c59cbf84fd9a074e8c60c5695257deecceacc27ae6 WatchSource:0}: Error finding container edff7fd72bac38fdaa2924c59cbf84fd9a074e8c60c5695257deecceacc27ae6: Status 404 returned error can't find the container with id edff7fd72bac38fdaa2924c59cbf84fd9a074e8c60c5695257deecceacc27ae6 Oct 30 17:11:51.203201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:51.203164 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22"] Oct 30 17:11:51.664997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:51.664881 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:51.783683 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:51.783645 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" event={"ID":"9875cf06-8438-4b8e-96b2-592d1ea387bf","Type":"ContainerStarted","Data":"1314121c0bebb10cf5e89b39b43f1226d204fad1e8bc2f3f3d1517f883568f2d"} Oct 30 17:11:51.784308 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:51.784280 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" event={"ID":"9875cf06-8438-4b8e-96b2-592d1ea387bf","Type":"ContainerStarted","Data":"edff7fd72bac38fdaa2924c59cbf84fd9a074e8c60c5695257deecceacc27ae6"} Oct 30 17:11:51.835961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:51.835880 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" podStartSLOduration=1.835865943 podStartE2EDuration="1.835865943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:51.834651888 +0000 UTC m=+1489.480162070" watchObservedRunningTime="2026-10-30 17:11:51.835865943 +0000 UTC m=+1489.481376120" Oct 30 17:11:51.845137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:51.845085 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:11:53.301828 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:53.301780 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:53.522091 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:53.522048 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:55.091326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:55.091149 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:56.197046 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:56.197002 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:56.783246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:56.783200 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:56.942629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:56.942587 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:57.464885 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:57.464849 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:57.932004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:57.931965 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw"] Oct 30 17:11:57.938468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:57.938438 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:57.974077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:57.974033 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b2c3fb-openshift-default-dockercfg-vbbf2\"" Oct 30 17:11:57.983082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:57.983047 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw"] Oct 30 17:11:58.047762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.047724 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:58.077519 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.077483 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-certs\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.077703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.077526 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-credential-socket\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.077703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.077620 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljvb4\" (UniqueName: \"kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-kube-api-access-ljvb4\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.077798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.077735 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-socket\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.077798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.077782 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9d4488-f73e-4096-bed3-603700716223-istiod-ca-cert\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.077888 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.077835 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9d4488-f73e-4096-bed3-603700716223-istio-podinfo\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.078095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.078069 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-data\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.078209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.078188 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-istio-token\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.078279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.078262 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-envoy\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.188059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.187198 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-certs\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.188059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.187256 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-credential-socket\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.188059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.187289 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljvb4\" (UniqueName: \"kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-kube-api-access-ljvb4\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.188059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.187371 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-socket\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.188059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.187420 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9d4488-f73e-4096-bed3-603700716223-istiod-ca-cert\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.188059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.187496 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9d4488-f73e-4096-bed3-603700716223-istio-podinfo\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.188059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.187606 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-data\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.188059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.187699 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-istio-token\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.188059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.187739 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-envoy\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.193918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.189309 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-socket\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.193918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.189580 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-certs\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.193918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.189834 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-credential-socket\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.193918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.190746 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-data\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.193918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.191560 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9d4488-f73e-4096-bed3-603700716223-istiod-ca-cert\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.214922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.196282 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-envoy\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.229022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.228593 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-istio-token\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.229022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.228606 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljvb4\" (UniqueName: \"kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-kube-api-access-ljvb4\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.257924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.245883 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9d4488-f73e-4096-bed3-603700716223-istio-podinfo\") pod \"e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.264929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.263249 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:11:58.880482 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:11:58.880437 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9d4488_f73e_4096_bed3_603700716223.slice/crio-2d4092c9e203680e6e7ab0f5addc68b00d8048457e348b02ef1abc9f948ddb7b WatchSource:0}: Error finding container 2d4092c9e203680e6e7ab0f5addc68b00d8048457e348b02ef1abc9f948ddb7b: Status 404 returned error can't find the container with id 2d4092c9e203680e6e7ab0f5addc68b00d8048457e348b02ef1abc9f948ddb7b Oct 30 17:11:58.881145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.881122 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw"] Oct 30 17:11:58.883708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.883677 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:58.883777 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.883741 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:58.883777 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.883771 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:11:58.943520 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:58.943488 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" event={"ID":"ba9d4488-f73e-4096-bed3-603700716223","Type":"ContainerStarted","Data":"2d4092c9e203680e6e7ab0f5addc68b00d8048457e348b02ef1abc9f948ddb7b"} Oct 30 17:11:59.167130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:59.165846 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:11:59.954211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:11:59.954169 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" event={"ID":"ba9d4488-f73e-4096-bed3-603700716223","Type":"ContainerStarted","Data":"16027ede312e132adb65d31837091d70f90ed6e202d897927844852178931959"} Oct 30 17:12:00.018545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:00.018476 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" podStartSLOduration=3.01845953 podStartE2EDuration="3.01845953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:00.015953481 +0000 UTC m=+1497.661463661" watchObservedRunningTime="2026-10-30 17:12:00.01845953 +0000 UTC m=+1497.663969712" Oct 30 17:12:00.270360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:00.268839 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:12:00.277431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:00.277403 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:12:00.772283 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:00.770950 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:00.846350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:00.846315 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:12:00.872657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:00.872622 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:12:00.963344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:00.963306 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:12:00.964453 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:00.964430 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:12:00.975241 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:00.975217 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:12:02.454710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:02.454666 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:02.659922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:02.659566 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:04.231913 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:04.231862 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:04.391109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:04.391075 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:04.883929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:04.862694 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:05.309400 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:05.309363 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:12:05.320528 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:05.320496 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:12:06.777926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:06.777673 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:08.239792 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:12:08.239754 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:12:08.784759 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:08.783735 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:12:08.996118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:08.995040 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:12:09.170971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:09.158023 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:09.172460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:09.172419 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:12:09.425738 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:09.425084 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:12:09.809984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:09.806599 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:12:09.809984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:09.807206 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:12:10.064307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:10.064189 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:12:10.070355 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:10.070179 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:12:10.848193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:10.848157 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:12:10.848678 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:10.848231 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:12:11.615231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:11.608840 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:13.348100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:13.348000 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:14.579101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:14.579060 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2"] Oct 30 17:12:14.580730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:14.579888 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" podUID="7c6b648f-9f86-4064-9890-8b653d7539b8" containerName="maas-api" containerID="cri-o://7a4e6a074469f8baeffd911d24cb6754bd22f2deaca54a61e42bcea44d3e79e6" gracePeriod=30 Oct 30 17:12:14.731872 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:14.731828 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:15.352457 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.350061 4828 generic.go:356] "Generic (PLEG): container finished" podID="7c6b648f-9f86-4064-9890-8b653d7539b8" containerID="7a4e6a074469f8baeffd911d24cb6754bd22f2deaca54a61e42bcea44d3e79e6" exitCode=0 Oct 30 17:12:15.352457 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.350405 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" event={"ID":"7c6b648f-9f86-4064-9890-8b653d7539b8","Type":"ContainerDied","Data":"7a4e6a074469f8baeffd911d24cb6754bd22f2deaca54a61e42bcea44d3e79e6"} Oct 30 17:12:15.352457 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.350465 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" event={"ID":"7c6b648f-9f86-4064-9890-8b653d7539b8","Type":"ContainerDied","Data":"806621f2ec9a5c5e1132f6f2d5614a00bb3070e06d1e6dabb698122978babeb8"} Oct 30 17:12:15.352457 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.350491 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="806621f2ec9a5c5e1132f6f2d5614a00bb3070e06d1e6dabb698122978babeb8" Oct 30 17:12:15.352457 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.350767 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:12:15.412986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.410021 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:15.530484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.530447 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6b648f-9f86-4064-9890-8b653d7539b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6b648f-9f86-4064-9890-8b653d7539b8-openshift-service-ca\") pod \"7c6b648f-9f86-4064-9890-8b653d7539b8\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " Oct 30 17:12:15.530779 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.530757 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6b648f-9f86-4064-9890-8b653d7539b8-kube-api-access-dr7r6\" (UniqueName: \"kubernetes.io/projected/7c6b648f-9f86-4064-9890-8b653d7539b8-kube-api-access-dr7r6\") pod \"7c6b648f-9f86-4064-9890-8b653d7539b8\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " Oct 30 17:12:15.531176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.531154 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-maas-api-tls\") pod \"7c6b648f-9f86-4064-9890-8b653d7539b8\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " Oct 30 17:12:15.531343 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.531327 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-metrics-certs\") pod \"7c6b648f-9f86-4064-9890-8b653d7539b8\" (UID: \"7c6b648f-9f86-4064-9890-8b653d7539b8\") " Oct 30 17:12:15.531819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.531526 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6b648f-9f86-4064-9890-8b653d7539b8-openshift-service-ca" pod "7c6b648f-9f86-4064-9890-8b653d7539b8" (UID: "7c6b648f-9f86-4064-9890-8b653d7539b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:15.532389 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.532365 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6b648f-9f86-4064-9890-8b653d7539b8-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:15.542714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.542675 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6b648f-9f86-4064-9890-8b653d7539b8-kube-api-access-dr7r6" pod "7c6b648f-9f86-4064-9890-8b653d7539b8" (UID: "7c6b648f-9f86-4064-9890-8b653d7539b8"). InnerVolumeSpecName "kube-api-access-dr7r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:15.543123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.543006 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-maas-api-tls" pod "7c6b648f-9f86-4064-9890-8b653d7539b8" (UID: "7c6b648f-9f86-4064-9890-8b653d7539b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:12:15.544103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.544074 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-metrics-certs" pod "7c6b648f-9f86-4064-9890-8b653d7539b8" (UID: "7c6b648f-9f86-4064-9890-8b653d7539b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:12:15.596797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.596692 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:15.636801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.636769 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:15.637125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.637107 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6b648f-9f86-4064-9890-8b653d7539b8-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:15.637218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.637132 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr7r6\" (UniqueName: \"kubernetes.io/projected/7c6b648f-9f86-4064-9890-8b653d7539b8-kube-api-access-dr7r6\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:16.000873 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:15.994600 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:16.239681 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:16.239642 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:16.360439 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:16.360406 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2" Oct 30 17:12:16.414128 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:12:16.414089 4828 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6b648f_9f86_4064_9890_8b653d7539b8.slice/crio-806621f2ec9a5c5e1132f6f2d5614a00bb3070e06d1e6dabb698122978babeb8\": RecentStats: unable to find data in memory cache]" Oct 30 17:12:16.431018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:16.430979 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2"] Oct 30 17:12:16.436307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:16.436273 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d1e5c-7888647b5c-kd6d2"] Oct 30 17:12:16.494162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:16.494114 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6b648f-9f86-4064-9890-8b653d7539b8" path="/var/lib/kubelet/pods/7c6b648f-9f86-4064-9890-8b653d7539b8/volumes" Oct 30 17:12:16.882893 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:16.882851 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:17.040160 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:17.040117 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:17.978975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:17.969542 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:18.387745 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:18.387710 4828 generic.go:356] "Generic (PLEG): container finished" podID="e2bbc703-7b2d-4c94-8730-63135103703e" containerID="01084f1d8eea4c1d0c123590df00d671ac525ab23d5407a94c53f030547f3d2a" exitCode=0 Oct 30 17:12:18.387964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:18.387779 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" event={"ID":"e2bbc703-7b2d-4c94-8730-63135103703e","Type":"ContainerDied","Data":"01084f1d8eea4c1d0c123590df00d671ac525ab23d5407a94c53f030547f3d2a"} Oct 30 17:12:18.388195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:18.388179 4828 scope.go:122] "RemoveContainer" containerID="01084f1d8eea4c1d0c123590df00d671ac525ab23d5407a94c53f030547f3d2a" Oct 30 17:12:18.390009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:18.389981 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:12:18.390112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:18.390055 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:12:18.390112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:18.390087 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:12:18.419261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:18.419239 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:12:19.417662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:19.417630 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" event={"ID":"e2bbc703-7b2d-4c94-8730-63135103703e","Type":"ContainerStarted","Data":"0aa541caae3a8feb1f8f5bbced6684411c2ea647a4d84eb2ff5cd368fc61cc06"} Oct 30 17:12:19.647791 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:19.647758 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:19.847934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:19.842524 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:12:19.847934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:19.842652 4828 patch_prober.go:29] interesting pod/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:12:19.847934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:19.842699 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:12:20.841597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:20.841559 4828 patch_prober.go:29] interesting pod/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:12:20.842047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:20.841635 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:12:21.644685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:21.644636 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:21.851925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:21.844698 4828 patch_prober.go:29] interesting pod/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:12:21.851925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:21.844772 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:12:22.440261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.440233 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:22.611794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.611758 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:22.737459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.737370 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph"] Oct 30 17:12:22.744327 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.744297 4828 cpu_manager.go:414] "containerMap: removing container" podUID="7c6b648f-9f86-4064-9890-8b653d7539b8" containerName="maas-api" Oct 30 17:12:22.744327 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.744327 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6b648f-9f86-4064-9890-8b653d7539b8" containerName="maas-api" Oct 30 17:12:22.744637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.744613 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6b648f-9f86-4064-9890-8b653d7539b8" containerName="maas-api" Oct 30 17:12:22.840484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.831346 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph"] Oct 30 17:12:22.840484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.831486 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:22.840484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.834328 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:22.845693 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.845635 4828 patch_prober.go:29] interesting pod/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:12:22.845926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.845713 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:12:22.851538 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.851503 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-39fecf\"/\"default-dockercfg-zrg79\"" Oct 30 17:12:22.864424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.864296 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-39fecf\"/\"openshift-service-ca.crt\"" Oct 30 17:12:22.864424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.864413 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-39fecf\"/\"test-model-39fecf-kserve-self-signed-certs\"" Oct 30 17:12:22.864849 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.864498 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-39fecf\"/\"kube-root-ca.crt\"" Oct 30 17:12:22.955085 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.955046 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-model-cache\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:22.955253 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.955099 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvzd6\" (UniqueName: \"kubernetes.io/projected/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-kube-api-access-vvzd6\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:22.955302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.955276 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-dshm\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:22.955336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.955310 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-home\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:22.955418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.955398 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tmp-dir\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:22.955580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:22.955568 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tls-certs\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.002133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.001847 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:23.087968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.060277 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tls-certs\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.089246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.089215 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-model-cache\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.089918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.089876 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvzd6\" (UniqueName: \"kubernetes.io/projected/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-kube-api-access-vvzd6\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.090670 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.090647 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-dshm\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.091173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.091154 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-home\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.091653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.091635 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tmp-dir\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.092013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.091546 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-home\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.092129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.089802 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-model-cache\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.092469 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.092450 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tmp-dir\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.119160 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.119115 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-dshm\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.148792 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.148644 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvzd6\" (UniqueName: \"kubernetes.io/projected/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-kube-api-access-vvzd6\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.160925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.151977 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tls-certs\") pod \"test-model-39fecf-kserve-77688688f5-wmtph\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.230927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.230232 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:23.841747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.841573 4828 patch_prober.go:29] interesting pod/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:12:23.841747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:23.841653 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:12:24.014005 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:12:24.013207 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a2f6b17_a32f_4a17_ba95_f2d5d62be9f3.slice/crio-26d6675fefc75385485fe51fec9a478ced4d29d9faeed7d6ae1352305c36df49 WatchSource:0}: Error finding container 26d6675fefc75385485fe51fec9a478ced4d29d9faeed7d6ae1352305c36df49: Status 404 returned error can't find the container with id 26d6675fefc75385485fe51fec9a478ced4d29d9faeed7d6ae1352305c36df49 Oct 30 17:12:24.014005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:24.013844 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph"] Oct 30 17:12:24.467274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:24.467152 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:24.649696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:24.649659 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" event={"ID":"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3","Type":"ContainerStarted","Data":"28e469177fca1c3dc43b50cb76a0c906d7976b742ad12fe3a16646b084ce053b"} Oct 30 17:12:24.649696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:24.649703 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" event={"ID":"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3","Type":"ContainerStarted","Data":"26d6675fefc75385485fe51fec9a478ced4d29d9faeed7d6ae1352305c36df49"} Oct 30 17:12:24.844824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:24.844189 4828 patch_prober.go:29] interesting pod/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 17:12:24.844824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:24.844255 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 17:12:24.959925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:24.941133 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" podStartSLOduration=2.941110194 podStartE2EDuration="2.941110194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:24.710337233 +0000 UTC m=+1522.355847413" watchObservedRunningTime="2026-10-30 17:12:24.941110194 +0000 UTC m=+1522.586620375" Oct 30 17:12:24.970928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:24.967967 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8659-bb37db"] Oct 30 17:12:25.050247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:25.050177 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8659-bb37db"] Oct 30 17:12:25.050737 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:25.050308 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8659-bb37db" Oct 30 17:12:25.147128 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:25.147024 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhm6x\" (UniqueName: \"kubernetes.io/projected/d9bfe7df-0ff9-46e6-ac2a-450a63e0998d-kube-api-access-fhm6x\") pod \"test-curl-8659-bb37db\" (UID: \"d9bfe7df-0ff9-46e6-ac2a-450a63e0998d\") " pod="opendatahub/test-curl-8659-bb37db" Oct 30 17:12:25.156244 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:25.156211 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wddsm\"" Oct 30 17:12:25.231238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:25.231204 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:25.248476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:25.248441 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhm6x\" (UniqueName: \"kubernetes.io/projected/d9bfe7df-0ff9-46e6-ac2a-450a63e0998d-kube-api-access-fhm6x\") pod \"test-curl-8659-bb37db\" (UID: \"d9bfe7df-0ff9-46e6-ac2a-450a63e0998d\") " pod="opendatahub/test-curl-8659-bb37db" Oct 30 17:12:25.290923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:25.280542 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhm6x\" (UniqueName: \"kubernetes.io/projected/d9bfe7df-0ff9-46e6-ac2a-450a63e0998d-kube-api-access-fhm6x\") pod \"test-curl-8659-bb37db\" (UID: \"d9bfe7df-0ff9-46e6-ac2a-450a63e0998d\") " pod="opendatahub/test-curl-8659-bb37db" Oct 30 17:12:25.367761 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:25.367730 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8659-bb37db" Oct 30 17:12:25.714728 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:25.714694 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8659-bb37db"] Oct 30 17:12:25.858984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:25.858954 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:12:25.859469 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:25.859447 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:12:25.860418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:25.860396 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:12:26.389810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.379174 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w"] Oct 30 17:12:26.389810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.386671 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.403476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.403444 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w"] Oct 30 17:12:26.404354 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.404329 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-db8417-serving-cert\"" Oct 30 17:12:26.471181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.471143 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.471385 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.471226 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.471521 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.471491 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7tvn\" (UniqueName: \"kubernetes.io/projected/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-kube-api-access-r7tvn\") pod \"maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.471735 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.471709 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.504343 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.504303 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:26.598925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.595490 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7tvn\" (UniqueName: \"kubernetes.io/projected/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-kube-api-access-r7tvn\") pod \"maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.598925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.595648 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.598925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.595809 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.598925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.595878 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.607852 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.607814 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.638959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.638926 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.642894 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.642342 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7tvn\" (UniqueName: \"kubernetes.io/projected/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-kube-api-access-r7tvn\") pod \"maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.664528 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.662983 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.724049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.724011 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:26.724535 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.724298 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:26.739742 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.735424 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8659-bb37db" event={"ID":"d9bfe7df-0ff9-46e6-ac2a-450a63e0998d","Type":"ContainerStarted","Data":"833e841a2fcee95340e4ecd453ff5ce387caacde4ff5788e211071de1684e76d"} Oct 30 17:12:26.739742 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.735485 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8659-bb37db" event={"ID":"d9bfe7df-0ff9-46e6-ac2a-450a63e0998d","Type":"ContainerStarted","Data":"313f84f951a2d2c43333e2749a4ddf243eb27ba7475edf28494ac628c082983d"} Oct 30 17:12:26.777789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:26.777737 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8659-bb37db" podStartSLOduration=2.777715374 podStartE2EDuration="2.777715374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:26.774540878 +0000 UTC m=+1524.420051058" watchObservedRunningTime="2026-10-30 17:12:26.777715374 +0000 UTC m=+1524.423225553" Oct 30 17:12:27.104339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:27.104303 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w"] Oct 30 17:12:27.105402 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:12:27.105373 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c7bccf_7a2e_464e_984f_8c6dc196d1c5.slice/crio-6f6b5f956cd8e70b1cb822f925ed59c9e0afb76df1439bafe0ecbea5c6876a53 WatchSource:0}: Error finding container 6f6b5f956cd8e70b1cb822f925ed59c9e0afb76df1439bafe0ecbea5c6876a53: Status 404 returned error can't find the container with id 6f6b5f956cd8e70b1cb822f925ed59c9e0afb76df1439bafe0ecbea5c6876a53 Oct 30 17:12:27.715284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:27.715249 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8659-bb37db"] Oct 30 17:12:27.727133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:27.727097 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8659-bb37db"] Oct 30 17:12:27.747164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:27.746831 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" event={"ID":"96c7bccf-7a2e-464e-984f-8c6dc196d1c5","Type":"ContainerStarted","Data":"886e826456b809ac9f85003b0ba5c4afb53a684d3f50f7634b72b086d43416d9"} Oct 30 17:12:27.747164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:27.746979 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" event={"ID":"96c7bccf-7a2e-464e-984f-8c6dc196d1c5","Type":"ContainerStarted","Data":"6f6b5f956cd8e70b1cb822f925ed59c9e0afb76df1439bafe0ecbea5c6876a53"} Oct 30 17:12:27.748513 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:27.747736 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:27.795095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:27.795029 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" podStartSLOduration=1.342487084 podStartE2EDuration="1.795007165s" totalImagesPullingTime="452.520081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:27.791864421 +0000 UTC m=+1525.437374602" watchObservedRunningTime="2026-10-30 17:12:27.795007165 +0000 UTC m=+1525.440517345" Oct 30 17:12:28.724924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:28.697830 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:28.821374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:28.821311 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8659-bb37db" podUID="d9bfe7df-0ff9-46e6-ac2a-450a63e0998d" containerName="test-curl-8659-bb37db" containerID="cri-o://833e841a2fcee95340e4ecd453ff5ce387caacde4ff5788e211071de1684e76d" gracePeriod=2 Oct 30 17:12:30.380924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:30.378347 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:31.098069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:31.098008 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8659-bb37db" Oct 30 17:12:31.242286 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:31.242251 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bfe7df-0ff9-46e6-ac2a-450a63e0998d-kube-api-access-fhm6x\" (UniqueName: \"kubernetes.io/projected/d9bfe7df-0ff9-46e6-ac2a-450a63e0998d-kube-api-access-fhm6x\") pod \"d9bfe7df-0ff9-46e6-ac2a-450a63e0998d\" (UID: \"d9bfe7df-0ff9-46e6-ac2a-450a63e0998d\") " Oct 30 17:12:31.244520 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:31.244492 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bfe7df-0ff9-46e6-ac2a-450a63e0998d-kube-api-access-fhm6x" pod "d9bfe7df-0ff9-46e6-ac2a-450a63e0998d" (UID: "d9bfe7df-0ff9-46e6-ac2a-450a63e0998d"). InnerVolumeSpecName "kube-api-access-fhm6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:31.344983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:31.344945 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhm6x\" (UniqueName: \"kubernetes.io/projected/d9bfe7df-0ff9-46e6-ac2a-450a63e0998d-kube-api-access-fhm6x\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:31.853144 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:31.853110 4828 generic.go:356] "Generic (PLEG): container finished" podID="d9bfe7df-0ff9-46e6-ac2a-450a63e0998d" containerID="833e841a2fcee95340e4ecd453ff5ce387caacde4ff5788e211071de1684e76d" exitCode=137 Oct 30 17:12:31.853685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:31.853154 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8659-bb37db" Oct 30 17:12:31.853685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:31.853209 4828 scope.go:122] "RemoveContainer" containerID="833e841a2fcee95340e4ecd453ff5ce387caacde4ff5788e211071de1684e76d" Oct 30 17:12:31.869324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:31.869289 4828 scope.go:122] "RemoveContainer" containerID="833e841a2fcee95340e4ecd453ff5ce387caacde4ff5788e211071de1684e76d" Oct 30 17:12:31.869718 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:12:31.869688 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833e841a2fcee95340e4ecd453ff5ce387caacde4ff5788e211071de1684e76d\": container with ID starting with 833e841a2fcee95340e4ecd453ff5ce387caacde4ff5788e211071de1684e76d not found: ID does not exist" containerID="833e841a2fcee95340e4ecd453ff5ce387caacde4ff5788e211071de1684e76d" Oct 30 17:12:31.869830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:31.869734 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833e841a2fcee95340e4ecd453ff5ce387caacde4ff5788e211071de1684e76d"} err="failed to get container status \"833e841a2fcee95340e4ecd453ff5ce387caacde4ff5788e211071de1684e76d\": rpc error: code = NotFound desc = could not find container \"833e841a2fcee95340e4ecd453ff5ce387caacde4ff5788e211071de1684e76d\": container with ID starting with 833e841a2fcee95340e4ecd453ff5ce387caacde4ff5788e211071de1684e76d not found: ID does not exist" Oct 30 17:12:31.876796 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:12:31.876767 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8659-bb37db\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="d9bfe7df-0ff9-46e6-ac2a-450a63e0998d" pod="opendatahub/test-curl-8659-bb37db" Oct 30 17:12:32.031927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:32.029014 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:32.494119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:32.494085 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9bfe7df-0ff9-46e6-ac2a-450a63e0998d" path="/var/lib/kubelet/pods/d9bfe7df-0ff9-46e6-ac2a-450a63e0998d/volumes" Oct 30 17:12:32.502888 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:12:32.502859 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8659-bb37db\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="d9bfe7df-0ff9-46e6-ac2a-450a63e0998d" pod="opendatahub/test-curl-8659-bb37db" Oct 30 17:12:33.153296 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:33.153255 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:33.231594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:33.231556 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:33.245277 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:33.245247 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:33.560441 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:33.560402 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn"] Oct 30 17:12:33.748974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:33.748936 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:33.891171 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:33.891087 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:12:33.911035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:33.910745 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:33.988923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:33.987611 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:12:35.559973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:35.558819 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:37.287931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:37.285623 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:37.918516 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:12:37.918475 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:12:38.083253 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:38.083219 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5"] Oct 30 17:12:38.266971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:38.266925 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:38.437937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:38.434036 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:38.561407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:38.561315 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" podUID="1cfbd42e-a9a9-4d16-b165-aa64bb079056" containerName="payload-processing" containerID="cri-o://a4e82e0555c02e98442e9d407c1ccbabeb1ace017853e00a076366e7e029df96" gracePeriod=25 Oct 30 17:12:39.100085 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.098307 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:39.156690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.156656 4828 generic.go:356] "Generic (PLEG): container finished" podID="1cfbd42e-a9a9-4d16-b165-aa64bb079056" containerID="a4e82e0555c02e98442e9d407c1ccbabeb1ace017853e00a076366e7e029df96" exitCode=0 Oct 30 17:12:39.157433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.157400 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" event={"ID":"1cfbd42e-a9a9-4d16-b165-aa64bb079056","Type":"ContainerDied","Data":"a4e82e0555c02e98442e9d407c1ccbabeb1ace017853e00a076366e7e029df96"} Oct 30 17:12:39.503491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.503213 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" Oct 30 17:12:39.549943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.533848 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cfbd42e-a9a9-4d16-b165-aa64bb079056-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfbd42e-a9a9-4d16-b165-aa64bb079056-plugins-config-volume\") pod \"1cfbd42e-a9a9-4d16-b165-aa64bb079056\" (UID: \"1cfbd42e-a9a9-4d16-b165-aa64bb079056\") " Oct 30 17:12:39.549943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.533982 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cfbd42e-a9a9-4d16-b165-aa64bb079056-kube-api-access-hq2s4\" (UniqueName: \"kubernetes.io/projected/1cfbd42e-a9a9-4d16-b165-aa64bb079056-kube-api-access-hq2s4\") pod \"1cfbd42e-a9a9-4d16-b165-aa64bb079056\" (UID: \"1cfbd42e-a9a9-4d16-b165-aa64bb079056\") " Oct 30 17:12:39.549943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.535394 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cfbd42e-a9a9-4d16-b165-aa64bb079056-plugins-config-volume" pod "1cfbd42e-a9a9-4d16-b165-aa64bb079056" (UID: "1cfbd42e-a9a9-4d16-b165-aa64bb079056"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:39.572535 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.569093 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfbd42e-a9a9-4d16-b165-aa64bb079056-kube-api-access-hq2s4" pod "1cfbd42e-a9a9-4d16-b165-aa64bb079056" (UID: "1cfbd42e-a9a9-4d16-b165-aa64bb079056"). InnerVolumeSpecName "kube-api-access-hq2s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:39.639507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.638085 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfbd42e-a9a9-4d16-b165-aa64bb079056-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:39.639507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.638125 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq2s4\" (UniqueName: \"kubernetes.io/projected/1cfbd42e-a9a9-4d16-b165-aa64bb079056-kube-api-access-hq2s4\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:39.650944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.643955 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc"] Oct 30 17:12:39.650944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.644231 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" podUID="748eedeb-7a6c-4bc9-b141-29be1431024a" containerName="istio-proxy" containerID="cri-o://78385a15ac2e4f1e16197a7ae49f17ff304512a0696e4f5fbab522505417732a" gracePeriod=30 Oct 30 17:12:39.683065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.682133 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6"] Oct 30 17:12:39.683065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.682476 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" podUID="08423875-76fe-41f6-995a-6c5832230d12" containerName="istio-proxy" containerID="cri-o://f352ae92bc51d628d40c4a30337cef2054019f29066e81152bcbb51b6b855f53" gracePeriod=30 Oct 30 17:12:39.729927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.726090 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx"] Oct 30 17:12:39.729927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.726342 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" podUID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" containerName="istio-proxy" containerID="cri-o://a2ae1caf0c6265b3b08f76d4322571415178a0cc5860dcae9fac7e651cb8effb" gracePeriod=30 Oct 30 17:12:39.820993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.816390 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj"] Oct 30 17:12:39.820993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.816791 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" containerID="cri-o://ba65c7e67874b7b51f098b04e588244995781d75c0a054b8af6c4c4ef9c4ddc4" gracePeriod=30 Oct 30 17:12:39.905992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:39.902390 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:40.063320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:40.063286 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:40.227527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:40.223144 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" event={"ID":"1cfbd42e-a9a9-4d16-b165-aa64bb079056","Type":"ContainerDied","Data":"3c3aa50d90b88a0c70891930e59f691d5dacfd96a50e19a5fe14583659d4ce79"} Oct 30 17:12:40.227527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:40.223222 4828 scope.go:122] "RemoveContainer" containerID="a4e82e0555c02e98442e9d407c1ccbabeb1ace017853e00a076366e7e029df96" Oct 30 17:12:40.227527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:40.223457 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn" Oct 30 17:12:40.297354 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:40.297319 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn"] Oct 30 17:12:40.302710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:40.302684 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-362bbf-65884ff9b6-cqbwn"] Oct 30 17:12:40.359084 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:40.359035 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:40.493003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:40.492919 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cfbd42e-a9a9-4d16-b165-aa64bb079056" path="/var/lib/kubelet/pods/1cfbd42e-a9a9-4d16-b165-aa64bb079056/volumes" Oct 30 17:12:40.681893 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:40.681837 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:42.177751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:42.177695 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:42.332924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:42.326351 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:42.730638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:42.730600 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:43.083665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:43.083615 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" podUID="5af96ea0-8c31-4d19-ac72-6d7083e590b2" containerName="payload-processing" containerID="cri-o://c423355c9d119e2771a459d44b30764e56b82fd410a230c471198340321ded43" gracePeriod=25 Oct 30 17:12:43.484857 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:43.484778 4828 generic.go:356] "Generic (PLEG): container finished" podID="5af96ea0-8c31-4d19-ac72-6d7083e590b2" containerID="c423355c9d119e2771a459d44b30764e56b82fd410a230c471198340321ded43" exitCode=0 Oct 30 17:12:43.485683 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:43.485404 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" event={"ID":"5af96ea0-8c31-4d19-ac72-6d7083e590b2","Type":"ContainerDied","Data":"c423355c9d119e2771a459d44b30764e56b82fd410a230c471198340321ded43"} Oct 30 17:12:43.655796 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:43.653649 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" Oct 30 17:12:43.745037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:43.744947 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af96ea0-8c31-4d19-ac72-6d7083e590b2-kube-api-access-kprwh\" (UniqueName: \"kubernetes.io/projected/5af96ea0-8c31-4d19-ac72-6d7083e590b2-kube-api-access-kprwh\") pod \"5af96ea0-8c31-4d19-ac72-6d7083e590b2\" (UID: \"5af96ea0-8c31-4d19-ac72-6d7083e590b2\") " Oct 30 17:12:43.745280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:43.745259 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af96ea0-8c31-4d19-ac72-6d7083e590b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af96ea0-8c31-4d19-ac72-6d7083e590b2-plugins-config-volume\") pod \"5af96ea0-8c31-4d19-ac72-6d7083e590b2\" (UID: \"5af96ea0-8c31-4d19-ac72-6d7083e590b2\") " Oct 30 17:12:43.745752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:43.745717 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af96ea0-8c31-4d19-ac72-6d7083e590b2-plugins-config-volume" pod "5af96ea0-8c31-4d19-ac72-6d7083e590b2" (UID: "5af96ea0-8c31-4d19-ac72-6d7083e590b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:43.746152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:43.746129 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af96ea0-8c31-4d19-ac72-6d7083e590b2-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:43.747883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:43.747854 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af96ea0-8c31-4d19-ac72-6d7083e590b2-kube-api-access-kprwh" pod "5af96ea0-8c31-4d19-ac72-6d7083e590b2" (UID: "5af96ea0-8c31-4d19-ac72-6d7083e590b2"). InnerVolumeSpecName "kube-api-access-kprwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:43.847746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:43.847710 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kprwh\" (UniqueName: \"kubernetes.io/projected/5af96ea0-8c31-4d19-ac72-6d7083e590b2-kube-api-access-kprwh\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:44.208422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:44.208385 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:44.381293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:44.381245 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:44.475167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:44.475084 4828 patch_prober.go:29] interesting pod/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:12:44.475317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:44.475167 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" podUID="748eedeb-7a6c-4bc9-b141-29be1431024a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:12:44.505715 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:44.505676 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" event={"ID":"5af96ea0-8c31-4d19-ac72-6d7083e590b2","Type":"ContainerDied","Data":"2530a480f29d4a6a8839baefdff343601149cffbe112f33561ea261c6c7f553f"} Oct 30 17:12:44.506167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:44.505758 4828 scope.go:122] "RemoveContainer" containerID="c423355c9d119e2771a459d44b30764e56b82fd410a230c471198340321ded43" Oct 30 17:12:44.506167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:44.505964 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5" Oct 30 17:12:44.579726 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:44.579687 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5"] Oct 30 17:12:44.590374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:44.590333 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ebeb6e1-6b4cbb6b94-f4ws5"] Oct 30 17:12:45.221836 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.221521 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:12:45.283365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.283324 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-istio-token\" (UniqueName: \"kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-istio-token\") pod \"08423875-76fe-41f6-995a-6c5832230d12\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " Oct 30 17:12:45.283520 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.283506 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-envoy\") pod \"08423875-76fe-41f6-995a-6c5832230d12\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " Oct 30 17:12:45.283644 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.283579 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08423875-76fe-41f6-995a-6c5832230d12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08423875-76fe-41f6-995a-6c5832230d12-istiod-ca-cert\") pod \"08423875-76fe-41f6-995a-6c5832230d12\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " Oct 30 17:12:45.283751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.283681 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08423875-76fe-41f6-995a-6c5832230d12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08423875-76fe-41f6-995a-6c5832230d12-istio-podinfo\") pod \"08423875-76fe-41f6-995a-6c5832230d12\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " Oct 30 17:12:45.283751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.283746 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-socket\") pod \"08423875-76fe-41f6-995a-6c5832230d12\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " Oct 30 17:12:45.283866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.283803 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-data\") pod \"08423875-76fe-41f6-995a-6c5832230d12\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " Oct 30 17:12:45.283934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.283863 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-kube-api-access-mz4xg\" (UniqueName: \"kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-kube-api-access-mz4xg\") pod \"08423875-76fe-41f6-995a-6c5832230d12\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " Oct 30 17:12:45.283934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.283913 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-certs\") pod \"08423875-76fe-41f6-995a-6c5832230d12\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " Oct 30 17:12:45.284096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.284045 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-credential-socket\") pod \"08423875-76fe-41f6-995a-6c5832230d12\" (UID: \"08423875-76fe-41f6-995a-6c5832230d12\") " Oct 30 17:12:45.287577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.287157 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-socket" pod "08423875-76fe-41f6-995a-6c5832230d12" (UID: "08423875-76fe-41f6-995a-6c5832230d12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.287577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.287528 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08423875-76fe-41f6-995a-6c5832230d12-istiod-ca-cert" pod "08423875-76fe-41f6-995a-6c5832230d12" (UID: "08423875-76fe-41f6-995a-6c5832230d12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:45.288189 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.287934 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-credential-socket" pod "08423875-76fe-41f6-995a-6c5832230d12" (UID: "08423875-76fe-41f6-995a-6c5832230d12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.288274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.288211 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-certs" pod "08423875-76fe-41f6-995a-6c5832230d12" (UID: "08423875-76fe-41f6-995a-6c5832230d12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.288274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.288265 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-data" pod "08423875-76fe-41f6-995a-6c5832230d12" (UID: "08423875-76fe-41f6-995a-6c5832230d12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.289476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.288883 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08423875-76fe-41f6-995a-6c5832230d12-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.289476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.288923 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.289476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.288937 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.289476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.288951 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.289476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.288964 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.304984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.304927 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-kube-api-access-mz4xg" pod "08423875-76fe-41f6-995a-6c5832230d12" (UID: "08423875-76fe-41f6-995a-6c5832230d12"). InnerVolumeSpecName "kube-api-access-mz4xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.306590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.306557 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-istio-token" pod "08423875-76fe-41f6-995a-6c5832230d12" (UID: "08423875-76fe-41f6-995a-6c5832230d12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.313082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.313034 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-envoy" pod "08423875-76fe-41f6-995a-6c5832230d12" (UID: "08423875-76fe-41f6-995a-6c5832230d12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.325413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.325376 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08423875-76fe-41f6-995a-6c5832230d12-istio-podinfo" pod "08423875-76fe-41f6-995a-6c5832230d12" (UID: "08423875-76fe-41f6-995a-6c5832230d12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:12:45.330614 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.330243 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 17:12:45.389480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.389444 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-certs\") pod \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " Oct 30 17:12:45.389680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.389527 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-credential-socket\") pod \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " Oct 30 17:12:45.389680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.389585 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istiod-ca-cert\") pod \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " Oct 30 17:12:45.389829 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.389740 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-envoy\") pod \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " Oct 30 17:12:45.389937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.389864 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-token\" (UniqueName: \"kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-token\") pod \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " Oct 30 17:12:45.389937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.389919 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-data\") pod \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " Oct 30 17:12:45.390062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.389995 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-socket\") pod \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " Oct 30 17:12:45.390062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.390036 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-podinfo\") pod \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " Oct 30 17:12:45.390173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.390124 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-credential-socket" pod "36000b4a-d11f-483e-9a73-e5bdf70daa5a" (UID: "36000b4a-d11f-483e-9a73-e5bdf70daa5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.390173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.390157 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-kube-api-access-64nl7\" (UniqueName: \"kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-kube-api-access-64nl7\") pod \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\" (UID: \"36000b4a-d11f-483e-9a73-e5bdf70daa5a\") " Oct 30 17:12:45.390597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.390504 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-certs" pod "36000b4a-d11f-483e-9a73-e5bdf70daa5a" (UID: "36000b4a-d11f-483e-9a73-e5bdf70daa5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.390742 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.390690 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istiod-ca-cert" pod "36000b4a-d11f-483e-9a73-e5bdf70daa5a" (UID: "36000b4a-d11f-483e-9a73-e5bdf70daa5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:45.391183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.390983 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-socket" pod "36000b4a-d11f-483e-9a73-e5bdf70daa5a" (UID: "36000b4a-d11f-483e-9a73-e5bdf70daa5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.391548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.391309 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08423875-76fe-41f6-995a-6c5832230d12-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.391548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.391332 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz4xg\" (UniqueName: \"kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-kube-api-access-mz4xg\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.391548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.391347 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.391548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.391362 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08423875-76fe-41f6-995a-6c5832230d12-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.391548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.391375 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.391548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.391425 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.391548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.391439 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.391548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.391451 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08423875-76fe-41f6-995a-6c5832230d12-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.392456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.392422 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-data" pod "36000b4a-d11f-483e-9a73-e5bdf70daa5a" (UID: "36000b4a-d11f-483e-9a73-e5bdf70daa5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.392780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.392760 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-token" pod "36000b4a-d11f-483e-9a73-e5bdf70daa5a" (UID: "36000b4a-d11f-483e-9a73-e5bdf70daa5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.395408 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.395373 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-envoy" pod "36000b4a-d11f-483e-9a73-e5bdf70daa5a" (UID: "36000b4a-d11f-483e-9a73-e5bdf70daa5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.396912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.396769 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-kube-api-access-64nl7" pod "36000b4a-d11f-483e-9a73-e5bdf70daa5a" (UID: "36000b4a-d11f-483e-9a73-e5bdf70daa5a"). InnerVolumeSpecName "kube-api-access-64nl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.399552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.399519 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-podinfo" pod "36000b4a-d11f-483e-9a73-e5bdf70daa5a" (UID: "36000b4a-d11f-483e-9a73-e5bdf70daa5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:12:45.420378 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.420042 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.492607 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-envoy\") pod \"748eedeb-7a6c-4bc9-b141-29be1431024a\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.492657 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-socket\") pod \"748eedeb-7a6c-4bc9-b141-29be1431024a\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.492737 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-certs\") pod \"748eedeb-7a6c-4bc9-b141-29be1431024a\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.492834 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-data\") pod \"748eedeb-7a6c-4bc9-b141-29be1431024a\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.493109 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-token\" (UniqueName: \"kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-token\") pod \"748eedeb-7a6c-4bc9-b141-29be1431024a\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.493142 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-podinfo\") pod \"748eedeb-7a6c-4bc9-b141-29be1431024a\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.493183 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-kube-api-access-f55vt\" (UniqueName: \"kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-kube-api-access-f55vt\") pod \"748eedeb-7a6c-4bc9-b141-29be1431024a\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.493264 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-credential-socket\") pod \"748eedeb-7a6c-4bc9-b141-29be1431024a\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.493294 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/748eedeb-7a6c-4bc9-b141-29be1431024a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748eedeb-7a6c-4bc9-b141-29be1431024a-istiod-ca-cert\") pod \"748eedeb-7a6c-4bc9-b141-29be1431024a\" (UID: \"748eedeb-7a6c-4bc9-b141-29be1431024a\") " Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.493959 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.493981 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64nl7\" (UniqueName: \"kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-kube-api-access-64nl7\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.493996 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.494010 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.494310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.494023 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36000b4a-d11f-483e-9a73-e5bdf70daa5a-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.498103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.498056 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748eedeb-7a6c-4bc9-b141-29be1431024a-istiod-ca-cert" pod "748eedeb-7a6c-4bc9-b141-29be1431024a" (UID: "748eedeb-7a6c-4bc9-b141-29be1431024a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:45.499690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.499648 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-socket" pod "748eedeb-7a6c-4bc9-b141-29be1431024a" (UID: "748eedeb-7a6c-4bc9-b141-29be1431024a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.499864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.499836 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-credential-socket" pod "748eedeb-7a6c-4bc9-b141-29be1431024a" (UID: "748eedeb-7a6c-4bc9-b141-29be1431024a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.500028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.499999 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-data" pod "748eedeb-7a6c-4bc9-b141-29be1431024a" (UID: "748eedeb-7a6c-4bc9-b141-29be1431024a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.500348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.500321 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-certs" pod "748eedeb-7a6c-4bc9-b141-29be1431024a" (UID: "748eedeb-7a6c-4bc9-b141-29be1431024a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.501762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.501719 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-podinfo" pod "748eedeb-7a6c-4bc9-b141-29be1431024a" (UID: "748eedeb-7a6c-4bc9-b141-29be1431024a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:12:45.502226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.502191 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-envoy" pod "748eedeb-7a6c-4bc9-b141-29be1431024a" (UID: "748eedeb-7a6c-4bc9-b141-29be1431024a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.503282 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.503262 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-token" pod "748eedeb-7a6c-4bc9-b141-29be1431024a" (UID: "748eedeb-7a6c-4bc9-b141-29be1431024a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.504024 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.503992 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-kube-api-access-f55vt" pod "748eedeb-7a6c-4bc9-b141-29be1431024a" (UID: "748eedeb-7a6c-4bc9-b141-29be1431024a"). InnerVolumeSpecName "kube-api-access-f55vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.524815 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.524784 4828 generic.go:356] "Generic (PLEG): container finished" podID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" containerID="a2ae1caf0c6265b3b08f76d4322571415178a0cc5860dcae9fac7e651cb8effb" exitCode=0 Oct 30 17:12:45.525212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.524981 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" event={"ID":"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf","Type":"ContainerDied","Data":"a2ae1caf0c6265b3b08f76d4322571415178a0cc5860dcae9fac7e651cb8effb"} Oct 30 17:12:45.528221 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.528204 4828 generic.go:356] "Generic (PLEG): container finished" podID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerID="ba65c7e67874b7b51f098b04e588244995781d75c0a054b8af6c4c4ef9c4ddc4" exitCode=0 Oct 30 17:12:45.528430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.528413 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" Oct 30 17:12:45.528515 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.528355 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" event={"ID":"36000b4a-d11f-483e-9a73-e5bdf70daa5a","Type":"ContainerDied","Data":"ba65c7e67874b7b51f098b04e588244995781d75c0a054b8af6c4c4ef9c4ddc4"} Oct 30 17:12:45.528567 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.528530 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj" event={"ID":"36000b4a-d11f-483e-9a73-e5bdf70daa5a","Type":"ContainerDied","Data":"28b32f9903441bb7e4a20f1f707cfabd3de3a4a84917657ec3f747918f99cd10"} Oct 30 17:12:45.528622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.528576 4828 scope.go:122] "RemoveContainer" containerID="ba65c7e67874b7b51f098b04e588244995781d75c0a054b8af6c4c4ef9c4ddc4" Oct 30 17:12:45.530713 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.530697 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" Oct 30 17:12:45.530802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.530736 4828 generic.go:356] "Generic (PLEG): container finished" podID="748eedeb-7a6c-4bc9-b141-29be1431024a" containerID="78385a15ac2e4f1e16197a7ae49f17ff304512a0696e4f5fbab522505417732a" exitCode=0 Oct 30 17:12:45.530802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.530751 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" event={"ID":"748eedeb-7a6c-4bc9-b141-29be1431024a","Type":"ContainerDied","Data":"78385a15ac2e4f1e16197a7ae49f17ff304512a0696e4f5fbab522505417732a"} Oct 30 17:12:45.530984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.530802 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc" event={"ID":"748eedeb-7a6c-4bc9-b141-29be1431024a","Type":"ContainerDied","Data":"05727e134a594f5df0a7340bd899dd27bdaaf3dc5cb02520896e2725dcd72792"} Oct 30 17:12:45.539786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.539763 4828 generic.go:356] "Generic (PLEG): container finished" podID="08423875-76fe-41f6-995a-6c5832230d12" containerID="f352ae92bc51d628d40c4a30337cef2054019f29066e81152bcbb51b6b855f53" exitCode=0 Oct 30 17:12:45.540090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.540072 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" event={"ID":"08423875-76fe-41f6-995a-6c5832230d12","Type":"ContainerDied","Data":"f352ae92bc51d628d40c4a30337cef2054019f29066e81152bcbb51b6b855f53"} Oct 30 17:12:45.540293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.540276 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" event={"ID":"08423875-76fe-41f6-995a-6c5832230d12","Type":"ContainerDied","Data":"ecfd8fe29f8f3f4af89d0ada96b0b9a1595c1bc69cb55b4d80b0ab723fcd4ccc"} Oct 30 17:12:45.540491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.540476 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6" Oct 30 17:12:45.568624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.568591 4828 scope.go:122] "RemoveContainer" containerID="a1381e78acbc22a2e8a2803e9bdc59c4ff74442edff9893a30ef23fbf2dc00e1" Oct 30 17:12:45.616482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.616455 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55vt\" (UniqueName: \"kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-kube-api-access-f55vt\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.616715 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.616698 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.616823 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.616809 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748eedeb-7a6c-4bc9-b141-29be1431024a-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.616920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.616908 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.617011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.617000 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.617090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.617080 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.617163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.617154 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.617240 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.617230 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.617312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.617303 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748eedeb-7a6c-4bc9-b141-29be1431024a-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.633148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.632647 4828 scope.go:122] "RemoveContainer" containerID="ba65c7e67874b7b51f098b04e588244995781d75c0a054b8af6c4c4ef9c4ddc4" Oct 30 17:12:45.633148 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:12:45.633028 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba65c7e67874b7b51f098b04e588244995781d75c0a054b8af6c4c4ef9c4ddc4\": container with ID starting with ba65c7e67874b7b51f098b04e588244995781d75c0a054b8af6c4c4ef9c4ddc4 not found: ID does not exist" containerID="ba65c7e67874b7b51f098b04e588244995781d75c0a054b8af6c4c4ef9c4ddc4" Oct 30 17:12:45.633148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.633060 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba65c7e67874b7b51f098b04e588244995781d75c0a054b8af6c4c4ef9c4ddc4"} err="failed to get container status \"ba65c7e67874b7b51f098b04e588244995781d75c0a054b8af6c4c4ef9c4ddc4\": rpc error: code = NotFound desc = could not find container \"ba65c7e67874b7b51f098b04e588244995781d75c0a054b8af6c4c4ef9c4ddc4\": container with ID starting with ba65c7e67874b7b51f098b04e588244995781d75c0a054b8af6c4c4ef9c4ddc4 not found: ID does not exist" Oct 30 17:12:45.633148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.633078 4828 scope.go:122] "RemoveContainer" containerID="a1381e78acbc22a2e8a2803e9bdc59c4ff74442edff9893a30ef23fbf2dc00e1" Oct 30 17:12:45.633397 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:12:45.633318 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1381e78acbc22a2e8a2803e9bdc59c4ff74442edff9893a30ef23fbf2dc00e1\": container with ID starting with a1381e78acbc22a2e8a2803e9bdc59c4ff74442edff9893a30ef23fbf2dc00e1 not found: ID does not exist" containerID="a1381e78acbc22a2e8a2803e9bdc59c4ff74442edff9893a30ef23fbf2dc00e1" Oct 30 17:12:45.633397 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.633335 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1381e78acbc22a2e8a2803e9bdc59c4ff74442edff9893a30ef23fbf2dc00e1"} err="failed to get container status \"a1381e78acbc22a2e8a2803e9bdc59c4ff74442edff9893a30ef23fbf2dc00e1\": rpc error: code = NotFound desc = could not find container \"a1381e78acbc22a2e8a2803e9bdc59c4ff74442edff9893a30ef23fbf2dc00e1\": container with ID starting with a1381e78acbc22a2e8a2803e9bdc59c4ff74442edff9893a30ef23fbf2dc00e1 not found: ID does not exist" Oct 30 17:12:45.633397 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.633350 4828 scope.go:122] "RemoveContainer" containerID="78385a15ac2e4f1e16197a7ae49f17ff304512a0696e4f5fbab522505417732a" Oct 30 17:12:45.640895 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.640870 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj"] Oct 30 17:12:45.651955 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.651937 4828 scope.go:122] "RemoveContainer" containerID="78385a15ac2e4f1e16197a7ae49f17ff304512a0696e4f5fbab522505417732a" Oct 30 17:12:45.652420 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:12:45.652398 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78385a15ac2e4f1e16197a7ae49f17ff304512a0696e4f5fbab522505417732a\": container with ID starting with 78385a15ac2e4f1e16197a7ae49f17ff304512a0696e4f5fbab522505417732a not found: ID does not exist" containerID="78385a15ac2e4f1e16197a7ae49f17ff304512a0696e4f5fbab522505417732a" Oct 30 17:12:45.652560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.652537 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78385a15ac2e4f1e16197a7ae49f17ff304512a0696e4f5fbab522505417732a"} err="failed to get container status \"78385a15ac2e4f1e16197a7ae49f17ff304512a0696e4f5fbab522505417732a\": rpc error: code = NotFound desc = could not find container \"78385a15ac2e4f1e16197a7ae49f17ff304512a0696e4f5fbab522505417732a\": container with ID starting with 78385a15ac2e4f1e16197a7ae49f17ff304512a0696e4f5fbab522505417732a not found: ID does not exist" Oct 30 17:12:45.652738 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.652724 4828 scope.go:122] "RemoveContainer" containerID="f352ae92bc51d628d40c4a30337cef2054019f29066e81152bcbb51b6b855f53" Oct 30 17:12:45.659287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.659265 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-64lxj"] Oct 30 17:12:45.666238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.666208 4828 scope.go:122] "RemoveContainer" containerID="f352ae92bc51d628d40c4a30337cef2054019f29066e81152bcbb51b6b855f53" Oct 30 17:12:45.666566 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:12:45.666539 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f352ae92bc51d628d40c4a30337cef2054019f29066e81152bcbb51b6b855f53\": container with ID starting with f352ae92bc51d628d40c4a30337cef2054019f29066e81152bcbb51b6b855f53 not found: ID does not exist" containerID="f352ae92bc51d628d40c4a30337cef2054019f29066e81152bcbb51b6b855f53" Oct 30 17:12:45.666656 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.666577 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f352ae92bc51d628d40c4a30337cef2054019f29066e81152bcbb51b6b855f53"} err="failed to get container status \"f352ae92bc51d628d40c4a30337cef2054019f29066e81152bcbb51b6b855f53\": rpc error: code = NotFound desc = could not find container \"f352ae92bc51d628d40c4a30337cef2054019f29066e81152bcbb51b6b855f53\": container with ID starting with f352ae92bc51d628d40c4a30337cef2054019f29066e81152bcbb51b6b855f53 not found: ID does not exist" Oct 30 17:12:45.690513 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.690463 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc"] Oct 30 17:12:45.701396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.701368 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-zvwtc"] Oct 30 17:12:45.732995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.732950 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6"] Oct 30 17:12:45.738242 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.738212 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-qgcm6"] Oct 30 17:12:45.746084 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.746020 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:12:45.821843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.821322 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-data\") pod \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " Oct 30 17:12:45.821843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.821430 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-token\" (UniqueName: \"kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-token\") pod \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " Oct 30 17:12:45.821843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.821472 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-podinfo\") pod \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " Oct 30 17:12:45.821843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.821521 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-kube-api-access-stq5d\" (UniqueName: \"kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-kube-api-access-stq5d\") pod \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " Oct 30 17:12:45.821843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.821602 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-envoy\") pod \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " Oct 30 17:12:45.821843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.821692 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istiod-ca-cert\") pod \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " Oct 30 17:12:45.821843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.821700 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-data" pod "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" (UID: "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.821843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.821827 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-socket\") pod \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " Oct 30 17:12:45.822426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.821867 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-certs\") pod \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " Oct 30 17:12:45.822426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.821969 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-credential-socket\") pod \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\" (UID: \"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf\") " Oct 30 17:12:45.822426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.822170 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-socket" pod "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" (UID: "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.822426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.822236 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istiod-ca-cert" pod "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" (UID: "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:45.822426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.822383 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-certs" pod "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" (UID: "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.822686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.822485 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-credential-socket" pod "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" (UID: "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.822764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.822742 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.822824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.822769 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.822824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.822784 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.822824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.822799 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.822824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.822812 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.826463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.826318 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-envoy" pod "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" (UID: "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:45.826463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.826337 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-token" pod "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" (UID: "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.826463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.826429 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-kube-api-access-stq5d" pod "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" (UID: "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf"). InnerVolumeSpecName "kube-api-access-stq5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:45.826870 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.826844 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-podinfo" pod "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" (UID: "a5baf94a-dc53-4365-ab04-2f7b0ad65bdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:12:45.923825 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.923788 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.923825 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.923819 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.923825 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.923830 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stq5d\" (UniqueName: \"kubernetes.io/projected/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-kube-api-access-stq5d\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:45.924096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:45.923841 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:12:46.027926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:46.020387 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:46.492708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:46.492609 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08423875-76fe-41f6-995a-6c5832230d12" path="/var/lib/kubelet/pods/08423875-76fe-41f6-995a-6c5832230d12/volumes" Oct 30 17:12:46.493300 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:46.493276 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" path="/var/lib/kubelet/pods/36000b4a-d11f-483e-9a73-e5bdf70daa5a/volumes" Oct 30 17:12:46.493807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:46.493786 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af96ea0-8c31-4d19-ac72-6d7083e590b2" path="/var/lib/kubelet/pods/5af96ea0-8c31-4d19-ac72-6d7083e590b2/volumes" Oct 30 17:12:46.494640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:46.494619 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="748eedeb-7a6c-4bc9-b141-29be1431024a" path="/var/lib/kubelet/pods/748eedeb-7a6c-4bc9-b141-29be1431024a/volumes" Oct 30 17:12:46.551097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:46.551067 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" Oct 30 17:12:46.551561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:46.551094 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx" event={"ID":"a5baf94a-dc53-4365-ab04-2f7b0ad65bdf","Type":"ContainerDied","Data":"e88b38e446d295188f52a6e0c834ba0223010b793cc7703838ab833f460653f7"} Oct 30 17:12:46.551561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:46.551163 4828 scope.go:122] "RemoveContainer" containerID="a2ae1caf0c6265b3b08f76d4322571415178a0cc5860dcae9fac7e651cb8effb" Oct 30 17:12:46.625716 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:46.625681 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx"] Oct 30 17:12:46.638319 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:46.638285 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4f406-openshift-default-55675b6897-6tsbx"] Oct 30 17:12:47.893641 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:47.893603 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:48.486247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:48.486211 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:48.525934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:48.525801 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" path="/var/lib/kubelet/pods/a5baf94a-dc53-4365-ab04-2f7b0ad65bdf/volumes" Oct 30 17:12:48.713524 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:48.713485 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:48.957483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:48.957442 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:50.116789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:50.116747 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:51.427299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:51.427257 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:52.464958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:52.447414 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:54.322928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:54.319739 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:56.475849 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:56.475793 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:56.639459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:56.639230 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:58.317933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:58.306053 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:58.354131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:58.354086 4828 patch_prober.go:29] interesting pod/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:12:58.354288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:58.354177 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:12:59.672093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:59.672054 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8"] Oct 30 17:12:59.672971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:59.672933 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" podUID="2760fc54-3612-4d2f-9e8d-c3054da972a9" containerName="maas-api" containerID="cri-o://517e58b10da5766f2b081e67ad9f9ea1da63cd85a25843bfcdced614f81403e2" gracePeriod=30 Oct 30 17:12:59.723711 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:59.723592 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:12:59.873074 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:12:59.873036 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:00.050249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.050205 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:13:00.136541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.136500 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2760fc54-3612-4d2f-9e8d-c3054da972a9-kube-api-access-pjxzk\" (UniqueName: \"kubernetes.io/projected/2760fc54-3612-4d2f-9e8d-c3054da972a9-kube-api-access-pjxzk\") pod \"2760fc54-3612-4d2f-9e8d-c3054da972a9\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " Oct 30 17:13:00.136757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.136670 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-maas-api-tls\") pod \"2760fc54-3612-4d2f-9e8d-c3054da972a9\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " Oct 30 17:13:00.136757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.136726 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-metrics-certs\") pod \"2760fc54-3612-4d2f-9e8d-c3054da972a9\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " Oct 30 17:13:00.137006 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.136860 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2760fc54-3612-4d2f-9e8d-c3054da972a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2760fc54-3612-4d2f-9e8d-c3054da972a9-openshift-service-ca\") pod \"2760fc54-3612-4d2f-9e8d-c3054da972a9\" (UID: \"2760fc54-3612-4d2f-9e8d-c3054da972a9\") " Oct 30 17:13:00.137972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.137938 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2760fc54-3612-4d2f-9e8d-c3054da972a9-openshift-service-ca" pod "2760fc54-3612-4d2f-9e8d-c3054da972a9" (UID: "2760fc54-3612-4d2f-9e8d-c3054da972a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:00.148854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.148532 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-maas-api-tls" pod "2760fc54-3612-4d2f-9e8d-c3054da972a9" (UID: "2760fc54-3612-4d2f-9e8d-c3054da972a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:00.148854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.148769 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2760fc54-3612-4d2f-9e8d-c3054da972a9-kube-api-access-pjxzk" pod "2760fc54-3612-4d2f-9e8d-c3054da972a9" (UID: "2760fc54-3612-4d2f-9e8d-c3054da972a9"). InnerVolumeSpecName "kube-api-access-pjxzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:00.160718 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.159054 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-metrics-certs" pod "2760fc54-3612-4d2f-9e8d-c3054da972a9" (UID: "2760fc54-3612-4d2f-9e8d-c3054da972a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:00.313645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.309570 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjxzk\" (UniqueName: \"kubernetes.io/projected/2760fc54-3612-4d2f-9e8d-c3054da972a9-kube-api-access-pjxzk\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:00.316264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.314436 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:00.318342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.317771 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:00.318342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.317852 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2760fc54-3612-4d2f-9e8d-c3054da972a9-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:00.318342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.317871 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2760fc54-3612-4d2f-9e8d-c3054da972a9-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:00.528414 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.527927 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:00.806557 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.806521 4828 generic.go:356] "Generic (PLEG): container finished" podID="2760fc54-3612-4d2f-9e8d-c3054da972a9" containerID="517e58b10da5766f2b081e67ad9f9ea1da63cd85a25843bfcdced614f81403e2" exitCode=0 Oct 30 17:13:00.807134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.806574 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" Oct 30 17:13:00.807134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.806601 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" event={"ID":"2760fc54-3612-4d2f-9e8d-c3054da972a9","Type":"ContainerDied","Data":"517e58b10da5766f2b081e67ad9f9ea1da63cd85a25843bfcdced614f81403e2"} Oct 30 17:13:00.807134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.806662 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8" event={"ID":"2760fc54-3612-4d2f-9e8d-c3054da972a9","Type":"ContainerDied","Data":"66e3d9cb9e27d8d2c162995bdea3d45654956caf78df5403d0a4438c8075bca5"} Oct 30 17:13:00.807134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.806693 4828 scope.go:122] "RemoveContainer" containerID="517e58b10da5766f2b081e67ad9f9ea1da63cd85a25843bfcdced614f81403e2" Oct 30 17:13:00.820753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.820731 4828 scope.go:122] "RemoveContainer" containerID="517e58b10da5766f2b081e67ad9f9ea1da63cd85a25843bfcdced614f81403e2" Oct 30 17:13:00.821205 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:00.821177 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517e58b10da5766f2b081e67ad9f9ea1da63cd85a25843bfcdced614f81403e2\": container with ID starting with 517e58b10da5766f2b081e67ad9f9ea1da63cd85a25843bfcdced614f81403e2 not found: ID does not exist" containerID="517e58b10da5766f2b081e67ad9f9ea1da63cd85a25843bfcdced614f81403e2" Oct 30 17:13:00.821299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.821218 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517e58b10da5766f2b081e67ad9f9ea1da63cd85a25843bfcdced614f81403e2"} err="failed to get container status \"517e58b10da5766f2b081e67ad9f9ea1da63cd85a25843bfcdced614f81403e2\": rpc error: code = NotFound desc = could not find container \"517e58b10da5766f2b081e67ad9f9ea1da63cd85a25843bfcdced614f81403e2\": container with ID starting with 517e58b10da5766f2b081e67ad9f9ea1da63cd85a25843bfcdced614f81403e2 not found: ID does not exist" Oct 30 17:13:00.863660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.863622 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8"] Oct 30 17:13:00.875610 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.875572 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7ebeb6e1-749dbc8898-rv5z8"] Oct 30 17:13:00.953303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:00.953202 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:01.275236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.272535 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": read tcp 10.132.0.2:35136->10.132.0.142:15021: read: connection reset by peer" start-of-body= Oct 30 17:13:01.275236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.272582 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": read tcp 10.132.0.2:35136->10.132.0.142:15021: read: connection reset by peer" Oct 30 17:13:01.818199 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.818170 4828 generic.go:356] "Generic (PLEG): container finished" podID="6f119ba5-b435-453d-809e-bb2795e3066d" containerID="e4cc0b0e5c1b171b36712679a8469dbc926573876fdf65a825863e48f106f20f" exitCode=0 Oct 30 17:13:01.818747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.818243 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" event={"ID":"6f119ba5-b435-453d-809e-bb2795e3066d","Type":"ContainerDied","Data":"e4cc0b0e5c1b171b36712679a8469dbc926573876fdf65a825863e48f106f20f"} Oct 30 17:13:01.818839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.818821 4828 scope.go:122] "RemoveContainer" containerID="e4cc0b0e5c1b171b36712679a8469dbc926573876fdf65a825863e48f106f20f" Oct 30 17:13:01.820100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.820078 4828 generic.go:356] "Generic (PLEG): container finished" podID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerID="861575c602f265de08be93b415495370836f5d8563d0b29fa688e288e560e1cc" exitCode=0 Oct 30 17:13:01.820234 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.820166 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" event={"ID":"3717264a-0933-41fa-9082-221ffa0c6ba5","Type":"ContainerDied","Data":"861575c602f265de08be93b415495370836f5d8563d0b29fa688e288e560e1cc"} Oct 30 17:13:01.820580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.820561 4828 scope.go:122] "RemoveContainer" containerID="861575c602f265de08be93b415495370836f5d8563d0b29fa688e288e560e1cc" Oct 30 17:13:01.820931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.820883 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:01.821048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.820970 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:01.821048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.821011 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:01.822303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.822277 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:01.822413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.822351 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:01.822413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:01.822402 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:02.502935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:02.501392 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2760fc54-3612-4d2f-9e8d-c3054da972a9" path="/var/lib/kubelet/pods/2760fc54-3612-4d2f-9e8d-c3054da972a9/volumes" Oct 30 17:13:02.599926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:02.595700 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:02.886052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:02.886008 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" event={"ID":"6f119ba5-b435-453d-809e-bb2795e3066d","Type":"ContainerStarted","Data":"e8b359a17219267c2d4c5c0c79203547bd088ca2ccd224f9264d8226ac735b85"} Oct 30 17:13:02.938019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:02.937970 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" event={"ID":"3717264a-0933-41fa-9082-221ffa0c6ba5","Type":"ContainerStarted","Data":"05afc1b30fb2fd94badffdaa00b9b271ad77979dcbeab81c61fd857c9c5af497"} Oct 30 17:13:03.114923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:03.114097 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:13:03.114923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:03.114877 4828 patch_prober.go:29] interesting pod/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 17:13:03.115196 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:03.114946 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 17:13:03.816925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:03.816453 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:13:03.816925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:03.816604 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 17:13:03.816925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:03.816649 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 17:13:04.122441 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:04.122342 4828 patch_prober.go:29] interesting pod/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 17:13:04.123070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:04.123037 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 17:13:04.810927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:04.807411 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 17:13:04.810927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:04.807478 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 17:13:04.845216 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:04.845179 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:05.139887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:05.139779 4828 patch_prober.go:29] interesting pod/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 17:13:05.139887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:05.139844 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 17:13:05.807861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:05.807822 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 17:13:05.808203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:05.808169 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 17:13:05.995941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:05.995837 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:06.130461 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:06.130364 4828 patch_prober.go:29] interesting pod/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 17:13:06.130733 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:06.130704 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 17:13:06.861220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:06.861182 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 17:13:06.861734 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:06.861687 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 17:13:07.141978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:07.141412 4828 patch_prober.go:29] interesting pod/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 17:13:07.141978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:07.141501 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 17:13:07.260813 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:07.250738 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:07.811208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:07.811170 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 17:13:07.811545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:07.811516 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 17:13:07.961541 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:07.961502 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:13:08.109459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:08.108826 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:08.118390 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:08.118347 4828 patch_prober.go:29] interesting pod/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 17:13:08.118680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:08.118646 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 17:13:09.045265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.045221 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8653-0e6800"] Oct 30 17:13:09.047266 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047232 4828 cpu_manager.go:414] "containerMap: removing container" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" Oct 30 17:13:09.047266 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047265 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" Oct 30 17:13:09.047472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047297 4828 cpu_manager.go:414] "containerMap: removing container" podUID="748eedeb-7a6c-4bc9-b141-29be1431024a" containerName="istio-proxy" Oct 30 17:13:09.047472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047306 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748eedeb-7a6c-4bc9-b141-29be1431024a" containerName="istio-proxy" Oct 30 17:13:09.047472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047347 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" containerName="istio-proxy" Oct 30 17:13:09.047472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047400 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" containerName="istio-proxy" Oct 30 17:13:09.047472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047426 4828 cpu_manager.go:414] "containerMap: removing container" podUID="08423875-76fe-41f6-995a-6c5832230d12" containerName="istio-proxy" Oct 30 17:13:09.047472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047434 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08423875-76fe-41f6-995a-6c5832230d12" containerName="istio-proxy" Oct 30 17:13:09.047472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047451 4828 cpu_manager.go:414] "containerMap: removing container" podUID="1cfbd42e-a9a9-4d16-b165-aa64bb079056" containerName="payload-processing" Oct 30 17:13:09.047472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047468 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cfbd42e-a9a9-4d16-b165-aa64bb079056" containerName="payload-processing" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047500 4828 cpu_manager.go:414] "containerMap: removing container" podUID="5af96ea0-8c31-4d19-ac72-6d7083e590b2" containerName="payload-processing" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047510 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af96ea0-8c31-4d19-ac72-6d7083e590b2" containerName="payload-processing" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047526 4828 cpu_manager.go:414] "containerMap: removing container" podUID="d9bfe7df-0ff9-46e6-ac2a-450a63e0998d" containerName="test-curl-8659-bb37db" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047533 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9bfe7df-0ff9-46e6-ac2a-450a63e0998d" containerName="test-curl-8659-bb37db" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047617 4828 cpu_manager.go:414] "containerMap: removing container" podUID="2760fc54-3612-4d2f-9e8d-c3054da972a9" containerName="maas-api" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047627 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2760fc54-3612-4d2f-9e8d-c3054da972a9" containerName="maas-api" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047810 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="748eedeb-7a6c-4bc9-b141-29be1431024a" containerName="istio-proxy" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047827 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5baf94a-dc53-4365-ab04-2f7b0ad65bdf" containerName="istio-proxy" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047841 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047849 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="08423875-76fe-41f6-995a-6c5832230d12" containerName="istio-proxy" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047859 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cfbd42e-a9a9-4d16-b165-aa64bb079056" containerName="payload-processing" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047870 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af96ea0-8c31-4d19-ac72-6d7083e590b2" containerName="payload-processing" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047881 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9bfe7df-0ff9-46e6-ac2a-450a63e0998d" containerName="test-curl-8659-bb37db" Oct 30 17:13:09.047912 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.047891 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="2760fc54-3612-4d2f-9e8d-c3054da972a9" containerName="maas-api" Oct 30 17:13:09.053241 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.053213 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-0e6800" Oct 30 17:13:09.073618 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.073566 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wddsm\"" Oct 30 17:13:09.093718 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.093673 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8653-0e6800"] Oct 30 17:13:09.146575 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.146539 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:13:09.148716 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.148570 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:13:09.151347 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.151255 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:13:09.180631 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.180156 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgkqp\" (UniqueName: \"kubernetes.io/projected/136dfc16-b4a0-4251-85dc-035024c06275-kube-api-access-lgkqp\") pod \"test-curl-8653-0e6800\" (UID: \"136dfc16-b4a0-4251-85dc-035024c06275\") " pod="opendatahub/test-curl-8653-0e6800" Oct 30 17:13:09.308494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.308399 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgkqp\" (UniqueName: \"kubernetes.io/projected/136dfc16-b4a0-4251-85dc-035024c06275-kube-api-access-lgkqp\") pod \"test-curl-8653-0e6800\" (UID: \"136dfc16-b4a0-4251-85dc-035024c06275\") " pod="opendatahub/test-curl-8653-0e6800" Oct 30 17:13:09.346925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.346630 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgkqp\" (UniqueName: \"kubernetes.io/projected/136dfc16-b4a0-4251-85dc-035024c06275-kube-api-access-lgkqp\") pod \"test-curl-8653-0e6800\" (UID: \"136dfc16-b4a0-4251-85dc-035024c06275\") " pod="opendatahub/test-curl-8653-0e6800" Oct 30 17:13:09.383037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.382836 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-0e6800" Oct 30 17:13:09.533514 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.531242 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:09.570080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.570001 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:13:09.571466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.571387 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:13:09.572558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.572538 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:13:09.871998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.868980 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8653-0e6800"] Oct 30 17:13:09.993947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.990330 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb"] Oct 30 17:13:09.993947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:09.990522 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" containerName="istio-proxy" containerID="cri-o://1c0721e85d29228503a3b3edb628e911bdd74667d78955875fa9dc675b8f20a3" gracePeriod=30 Oct 30 17:13:10.160108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:10.160009 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8653-0e6800" event={"ID":"136dfc16-b4a0-4251-85dc-035024c06275","Type":"ContainerStarted","Data":"f369dc377cc2669fd46bf8ed49e33c4940b438c1dc5816daff8754d97b61122e"} Oct 30 17:13:10.160108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:10.160064 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8653-0e6800" event={"ID":"136dfc16-b4a0-4251-85dc-035024c06275","Type":"ContainerStarted","Data":"a0b0dbd5bbd3261654c82f547613c29d80f2feafa06c9858aa63d5f08c710144"} Oct 30 17:13:10.205184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:10.205120 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8653-0e6800" podStartSLOduration=1.205102703 podStartE2EDuration="1.205102703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:10.199729162 +0000 UTC m=+1567.845239342" watchObservedRunningTime="2026-10-30 17:13:10.205102703 +0000 UTC m=+1567.850612884" Oct 30 17:13:11.098530 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.098482 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8653-0e6800"] Oct 30 17:13:11.106238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.106053 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8653-0e6800"] Oct 30 17:13:11.296197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.292809 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:11.342344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.341943 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8653-0aba40"] Oct 30 17:13:11.343222 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.343189 4828 cpu_manager.go:414] "containerMap: removing container" podUID="136dfc16-b4a0-4251-85dc-035024c06275" containerName="test-curl-8653-0e6800" Oct 30 17:13:11.343222 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.343221 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136dfc16-b4a0-4251-85dc-035024c06275" containerName="test-curl-8653-0e6800" Oct 30 17:13:11.343222 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.343263 4828 cpu_manager.go:414] "containerMap: removing container" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" Oct 30 17:13:11.343222 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.343271 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" Oct 30 17:13:11.343721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.343552 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="36000b4a-d11f-483e-9a73-e5bdf70daa5a" containerName="istio-proxy" Oct 30 17:13:11.343721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.343577 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="136dfc16-b4a0-4251-85dc-035024c06275" containerName="test-curl-8653-0e6800" Oct 30 17:13:11.355359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.355279 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-0aba40" Oct 30 17:13:11.363399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.363368 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8653-0aba40"] Oct 30 17:13:11.413475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.413433 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v88m\" (UniqueName: \"kubernetes.io/projected/e4aa1cf3-5db3-4a4d-800d-e95249b49147-kube-api-access-2v88m\") pod \"test-curl-8653-0aba40\" (UID: \"e4aa1cf3-5db3-4a4d-800d-e95249b49147\") " pod="opendatahub/test-curl-8653-0aba40" Oct 30 17:13:11.514758 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.514719 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v88m\" (UniqueName: \"kubernetes.io/projected/e4aa1cf3-5db3-4a4d-800d-e95249b49147-kube-api-access-2v88m\") pod \"test-curl-8653-0aba40\" (UID: \"e4aa1cf3-5db3-4a4d-800d-e95249b49147\") " pod="opendatahub/test-curl-8653-0aba40" Oct 30 17:13:11.539548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.539511 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v88m\" (UniqueName: \"kubernetes.io/projected/e4aa1cf3-5db3-4a4d-800d-e95249b49147-kube-api-access-2v88m\") pod \"test-curl-8653-0aba40\" (UID: \"e4aa1cf3-5db3-4a4d-800d-e95249b49147\") " pod="opendatahub/test-curl-8653-0aba40" Oct 30 17:13:11.680782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.680699 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-0aba40" Oct 30 17:13:11.991424 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:13:11.991374 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4aa1cf3_5db3_4a4d_800d_e95249b49147.slice/crio-52b611a89da3daf83969a77cf8c85734b4901719bd4f74f68e2f492354ec6828 WatchSource:0}: Error finding container 52b611a89da3daf83969a77cf8c85734b4901719bd4f74f68e2f492354ec6828: Status 404 returned error can't find the container with id 52b611a89da3daf83969a77cf8c85734b4901719bd4f74f68e2f492354ec6828 Oct 30 17:13:11.993709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:11.993675 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8653-0aba40"] Oct 30 17:13:12.181519 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:12.181480 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8653-0aba40" event={"ID":"e4aa1cf3-5db3-4a4d-800d-e95249b49147","Type":"ContainerStarted","Data":"2274afd804dea021044c66b68240000f8e0585e753988f64451a84e16ed8fa34"} Oct 30 17:13:12.181696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:12.181534 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8653-0aba40" event={"ID":"e4aa1cf3-5db3-4a4d-800d-e95249b49147","Type":"ContainerStarted","Data":"52b611a89da3daf83969a77cf8c85734b4901719bd4f74f68e2f492354ec6828"} Oct 30 17:13:12.181757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:12.181712 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8653-0e6800" podUID="136dfc16-b4a0-4251-85dc-035024c06275" containerName="test-curl-8653-0e6800" containerID="cri-o://f369dc377cc2669fd46bf8ed49e33c4940b438c1dc5816daff8754d97b61122e" gracePeriod=2 Oct 30 17:13:12.222007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:12.221949 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8653-0aba40" podStartSLOduration=1.2219309 podStartE2EDuration="1.2219309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:12.220093333 +0000 UTC m=+1569.865603512" watchObservedRunningTime="2026-10-30 17:13:12.2219309 +0000 UTC m=+1569.867441078" Oct 30 17:13:12.234178 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:12.234142 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8653-0e6800\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="136dfc16-b4a0-4251-85dc-035024c06275" pod="opendatahub/test-curl-8653-0e6800" Oct 30 17:13:12.502780 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:12.502741 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8653-0e6800\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="136dfc16-b4a0-4251-85dc-035024c06275" pod="opendatahub/test-curl-8653-0e6800" Oct 30 17:13:12.936052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:12.936010 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:13.234363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:13.234222 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8653-0aba40"] Oct 30 17:13:13.240703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:13.240668 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8653-0aba40"] Oct 30 17:13:13.388397 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:13.388363 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8653-868444"] Oct 30 17:13:13.389599 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:13.389578 4828 cpu_manager.go:414] "containerMap: removing container" podUID="e4aa1cf3-5db3-4a4d-800d-e95249b49147" containerName="test-curl-8653-0aba40" Oct 30 17:13:13.389654 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:13.389603 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4aa1cf3-5db3-4a4d-800d-e95249b49147" containerName="test-curl-8653-0aba40" Oct 30 17:13:13.389821 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:13.389808 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4aa1cf3-5db3-4a4d-800d-e95249b49147" containerName="test-curl-8653-0aba40" Oct 30 17:13:13.393310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:13.393288 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-868444" Oct 30 17:13:13.411017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:13.410808 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8653-868444"] Oct 30 17:13:13.451559 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:13.451516 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlv99\" (UniqueName: \"kubernetes.io/projected/7eb2c034-c707-4550-81e3-cadb457fabcd-kube-api-access-mlv99\") pod \"test-curl-8653-868444\" (UID: \"7eb2c034-c707-4550-81e3-cadb457fabcd\") " pod="opendatahub/test-curl-8653-868444" Oct 30 17:13:13.553283 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:13.553237 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlv99\" (UniqueName: \"kubernetes.io/projected/7eb2c034-c707-4550-81e3-cadb457fabcd-kube-api-access-mlv99\") pod \"test-curl-8653-868444\" (UID: \"7eb2c034-c707-4550-81e3-cadb457fabcd\") " pod="opendatahub/test-curl-8653-868444" Oct 30 17:13:13.577562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:13.577530 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlv99\" (UniqueName: \"kubernetes.io/projected/7eb2c034-c707-4550-81e3-cadb457fabcd-kube-api-access-mlv99\") pod \"test-curl-8653-868444\" (UID: \"7eb2c034-c707-4550-81e3-cadb457fabcd\") " pod="opendatahub/test-curl-8653-868444" Oct 30 17:13:13.710425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:13.710391 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-868444" Oct 30 17:13:14.023194 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:13:14.023159 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7eb2c034_c707_4550_81e3_cadb457fabcd.slice/crio-7d6ba05524d974a90414dd4b6bf803e4e490cd6c2803c8e4aecd4e17f381422b WatchSource:0}: Error finding container 7d6ba05524d974a90414dd4b6bf803e4e490cd6c2803c8e4aecd4e17f381422b: Status 404 returned error can't find the container with id 7d6ba05524d974a90414dd4b6bf803e4e490cd6c2803c8e4aecd4e17f381422b Oct 30 17:13:14.023577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:14.023556 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8653-868444"] Oct 30 17:13:14.202959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:14.202929 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8653-868444" event={"ID":"7eb2c034-c707-4550-81e3-cadb457fabcd","Type":"ContainerStarted","Data":"63bf837f1e477e4895f59e69fa45135a7925a9843b5d1c2b64ebbfb6570b9940"} Oct 30 17:13:14.203101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:14.202975 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8653-868444" event={"ID":"7eb2c034-c707-4550-81e3-cadb457fabcd","Type":"ContainerStarted","Data":"7d6ba05524d974a90414dd4b6bf803e4e490cd6c2803c8e4aecd4e17f381422b"} Oct 30 17:13:14.203101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:14.203055 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8653-0aba40" podUID="e4aa1cf3-5db3-4a4d-800d-e95249b49147" containerName="test-curl-8653-0aba40" containerID="cri-o://2274afd804dea021044c66b68240000f8e0585e753988f64451a84e16ed8fa34" gracePeriod=2 Oct 30 17:13:14.217340 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:14.217254 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8653-0aba40\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="e4aa1cf3-5db3-4a4d-800d-e95249b49147" pod="opendatahub/test-curl-8653-0aba40" Oct 30 17:13:14.261224 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:14.261171 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8653-868444" podStartSLOduration=1.261155866 podStartE2EDuration="1.261155866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:14.258786732 +0000 UTC m=+1571.904296911" watchObservedRunningTime="2026-10-30 17:13:14.261155866 +0000 UTC m=+1571.906666045" Oct 30 17:13:14.272943 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:14.272828 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8653-0aba40\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="e4aa1cf3-5db3-4a4d-800d-e95249b49147" pod="opendatahub/test-curl-8653-0aba40" Oct 30 17:13:14.398326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:14.398301 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-0e6800" Oct 30 17:13:14.412431 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:14.412380 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8653-0aba40\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="e4aa1cf3-5db3-4a4d-800d-e95249b49147" pod="opendatahub/test-curl-8653-0aba40" Oct 30 17:13:14.467579 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:14.467541 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136dfc16-b4a0-4251-85dc-035024c06275-kube-api-access-lgkqp\" (UniqueName: \"kubernetes.io/projected/136dfc16-b4a0-4251-85dc-035024c06275-kube-api-access-lgkqp\") pod \"136dfc16-b4a0-4251-85dc-035024c06275\" (UID: \"136dfc16-b4a0-4251-85dc-035024c06275\") " Oct 30 17:13:14.469816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:14.469785 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136dfc16-b4a0-4251-85dc-035024c06275-kube-api-access-lgkqp" pod "136dfc16-b4a0-4251-85dc-035024c06275" (UID: "136dfc16-b4a0-4251-85dc-035024c06275"). InnerVolumeSpecName "kube-api-access-lgkqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:14.496999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:14.496963 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136dfc16-b4a0-4251-85dc-035024c06275" path="/var/lib/kubelet/pods/136dfc16-b4a0-4251-85dc-035024c06275/volumes" Oct 30 17:13:14.570983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:14.570870 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgkqp\" (UniqueName: \"kubernetes.io/projected/136dfc16-b4a0-4251-85dc-035024c06275-kube-api-access-lgkqp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:14.684270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:14.684225 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:15.174999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.174965 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8653-868444"] Oct 30 17:13:15.180273 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.180239 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8653-868444"] Oct 30 17:13:15.219583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.219544 4828 generic.go:356] "Generic (PLEG): container finished" podID="136dfc16-b4a0-4251-85dc-035024c06275" containerID="f369dc377cc2669fd46bf8ed49e33c4940b438c1dc5816daff8754d97b61122e" exitCode=137 Oct 30 17:13:15.219784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.219696 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-0e6800" Oct 30 17:13:15.219784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.219775 4828 scope.go:122] "RemoveContainer" containerID="f369dc377cc2669fd46bf8ed49e33c4940b438c1dc5816daff8754d97b61122e" Oct 30 17:13:15.222090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.222067 4828 generic.go:356] "Generic (PLEG): container finished" podID="008aa481-f9ff-4742-b198-d8556de19b3a" containerID="1c0721e85d29228503a3b3edb628e911bdd74667d78955875fa9dc675b8f20a3" exitCode=0 Oct 30 17:13:15.222743 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.222718 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" event={"ID":"008aa481-f9ff-4742-b198-d8556de19b3a","Type":"ContainerDied","Data":"1c0721e85d29228503a3b3edb628e911bdd74667d78955875fa9dc675b8f20a3"} Oct 30 17:13:15.234302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.234277 4828 scope.go:122] "RemoveContainer" containerID="f369dc377cc2669fd46bf8ed49e33c4940b438c1dc5816daff8754d97b61122e" Oct 30 17:13:15.234811 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:15.234773 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f369dc377cc2669fd46bf8ed49e33c4940b438c1dc5816daff8754d97b61122e\": container with ID starting with f369dc377cc2669fd46bf8ed49e33c4940b438c1dc5816daff8754d97b61122e not found: ID does not exist" containerID="f369dc377cc2669fd46bf8ed49e33c4940b438c1dc5816daff8754d97b61122e" Oct 30 17:13:15.235254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.234822 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f369dc377cc2669fd46bf8ed49e33c4940b438c1dc5816daff8754d97b61122e"} err="failed to get container status \"f369dc377cc2669fd46bf8ed49e33c4940b438c1dc5816daff8754d97b61122e\": rpc error: code = NotFound desc = could not find container \"f369dc377cc2669fd46bf8ed49e33c4940b438c1dc5816daff8754d97b61122e\": container with ID starting with f369dc377cc2669fd46bf8ed49e33c4940b438c1dc5816daff8754d97b61122e not found: ID does not exist" Oct 30 17:13:15.335272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.335003 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8653-c4fb43"] Oct 30 17:13:15.336089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.336065 4828 cpu_manager.go:414] "containerMap: removing container" podUID="7eb2c034-c707-4550-81e3-cadb457fabcd" containerName="test-curl-8653-868444" Oct 30 17:13:15.336089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.336088 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb2c034-c707-4550-81e3-cadb457fabcd" containerName="test-curl-8653-868444" Oct 30 17:13:15.336234 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.336223 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb2c034-c707-4550-81e3-cadb457fabcd" containerName="test-curl-8653-868444" Oct 30 17:13:15.339765 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.339746 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-c4fb43" Oct 30 17:13:15.354813 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.354733 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8653-c4fb43"] Oct 30 17:13:15.521946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.521915 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shtgk\" (UniqueName: \"kubernetes.io/projected/0b2c9fd9-f302-4d62-af98-a771e28d75d6-kube-api-access-shtgk\") pod \"test-curl-8653-c4fb43\" (UID: \"0b2c9fd9-f302-4d62-af98-a771e28d75d6\") " pod="opendatahub/test-curl-8653-c4fb43" Oct 30 17:13:15.623777 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.623553 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtgk\" (UniqueName: \"kubernetes.io/projected/0b2c9fd9-f302-4d62-af98-a771e28d75d6-kube-api-access-shtgk\") pod \"test-curl-8653-c4fb43\" (UID: \"0b2c9fd9-f302-4d62-af98-a771e28d75d6\") " pod="opendatahub/test-curl-8653-c4fb43" Oct 30 17:13:15.658528 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.658481 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shtgk\" (UniqueName: \"kubernetes.io/projected/0b2c9fd9-f302-4d62-af98-a771e28d75d6-kube-api-access-shtgk\") pod \"test-curl-8653-c4fb43\" (UID: \"0b2c9fd9-f302-4d62-af98-a771e28d75d6\") " pod="opendatahub/test-curl-8653-c4fb43" Oct 30 17:13:15.698576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.698543 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:13:15.724996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.724961 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-credential-socket\") pod \"008aa481-f9ff-4742-b198-d8556de19b3a\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " Oct 30 17:13:15.724996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.725004 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-data\") pod \"008aa481-f9ff-4742-b198-d8556de19b3a\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " Oct 30 17:13:15.725217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.725086 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/008aa481-f9ff-4742-b198-d8556de19b3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008aa481-f9ff-4742-b198-d8556de19b3a-istio-podinfo\") pod \"008aa481-f9ff-4742-b198-d8556de19b3a\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " Oct 30 17:13:15.725217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.725161 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-certs\") pod \"008aa481-f9ff-4742-b198-d8556de19b3a\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " Oct 30 17:13:15.725305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.725238 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/008aa481-f9ff-4742-b198-d8556de19b3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008aa481-f9ff-4742-b198-d8556de19b3a-istiod-ca-cert\") pod \"008aa481-f9ff-4742-b198-d8556de19b3a\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " Oct 30 17:13:15.725305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.725289 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-envoy\") pod \"008aa481-f9ff-4742-b198-d8556de19b3a\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " Oct 30 17:13:15.725404 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.725304 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-data" pod "008aa481-f9ff-4742-b198-d8556de19b3a" (UID: "008aa481-f9ff-4742-b198-d8556de19b3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:15.725404 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.725314 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-credential-socket" pod "008aa481-f9ff-4742-b198-d8556de19b3a" (UID: "008aa481-f9ff-4742-b198-d8556de19b3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:15.725404 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.725375 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-socket\") pod \"008aa481-f9ff-4742-b198-d8556de19b3a\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " Oct 30 17:13:15.725551 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.725463 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-istio-token\" (UniqueName: \"kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-istio-token\") pod \"008aa481-f9ff-4742-b198-d8556de19b3a\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " Oct 30 17:13:15.725551 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.725501 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-certs" pod "008aa481-f9ff-4742-b198-d8556de19b3a" (UID: "008aa481-f9ff-4742-b198-d8556de19b3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:15.725657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.725560 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-kube-api-access-vlhls\" (UniqueName: \"kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-kube-api-access-vlhls\") pod \"008aa481-f9ff-4742-b198-d8556de19b3a\" (UID: \"008aa481-f9ff-4742-b198-d8556de19b3a\") " Oct 30 17:13:15.725714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.725678 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-socket" pod "008aa481-f9ff-4742-b198-d8556de19b3a" (UID: "008aa481-f9ff-4742-b198-d8556de19b3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:15.726454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.726062 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008aa481-f9ff-4742-b198-d8556de19b3a-istiod-ca-cert" pod "008aa481-f9ff-4742-b198-d8556de19b3a" (UID: "008aa481-f9ff-4742-b198-d8556de19b3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:15.726454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.726392 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.726454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.726410 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008aa481-f9ff-4742-b198-d8556de19b3a-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.726454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.726420 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.726454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.726428 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.726454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.726437 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.728023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.727922 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-envoy" pod "008aa481-f9ff-4742-b198-d8556de19b3a" (UID: "008aa481-f9ff-4742-b198-d8556de19b3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:15.728390 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.728364 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/008aa481-f9ff-4742-b198-d8556de19b3a-istio-podinfo" pod "008aa481-f9ff-4742-b198-d8556de19b3a" (UID: "008aa481-f9ff-4742-b198-d8556de19b3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:13:15.728390 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.728377 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-kube-api-access-vlhls" pod "008aa481-f9ff-4742-b198-d8556de19b3a" (UID: "008aa481-f9ff-4742-b198-d8556de19b3a"). InnerVolumeSpecName "kube-api-access-vlhls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:15.728970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.728949 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-istio-token" pod "008aa481-f9ff-4742-b198-d8556de19b3a" (UID: "008aa481-f9ff-4742-b198-d8556de19b3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:15.828069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.828033 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008aa481-f9ff-4742-b198-d8556de19b3a-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.828069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.828064 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.828069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.828074 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlhls\" (UniqueName: \"kubernetes.io/projected/008aa481-f9ff-4742-b198-d8556de19b3a-kube-api-access-vlhls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.828309 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.828084 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008aa481-f9ff-4742-b198-d8556de19b3a-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:15.954188 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:15.954098 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-c4fb43" Oct 30 17:13:16.182996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:16.182944 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g"] Oct 30 17:13:16.183241 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:16.183211 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" podUID="2459de27-81ba-4d49-886e-8629d8cdbf60" containerName="maas-api" containerID="cri-o://5a80a93444f37aa6bad98f9f0dbe0b59e797cf81ba9ce1d4ad58a5ad27441a86" gracePeriod=30 Oct 30 17:13:16.242457 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:16.242430 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" Oct 30 17:13:16.242603 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:16.242465 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb" event={"ID":"008aa481-f9ff-4742-b198-d8556de19b3a","Type":"ContainerDied","Data":"22e631e7f640d121b881f018f3b22a525b0bd0f5bfe53fd9e02c214c968551a6"} Oct 30 17:13:16.242603 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:16.242535 4828 scope.go:122] "RemoveContainer" containerID="1c0721e85d29228503a3b3edb628e911bdd74667d78955875fa9dc675b8f20a3" Oct 30 17:13:16.242718 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:16.242662 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8653-868444" podUID="7eb2c034-c707-4550-81e3-cadb457fabcd" containerName="test-curl-8653-868444" containerID="cri-o://63bf837f1e477e4895f59e69fa45135a7925a9843b5d1c2b64ebbfb6570b9940" gracePeriod=2 Oct 30 17:13:16.273177 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:16.273146 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8653-868444\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="7eb2c034-c707-4550-81e3-cadb457fabcd" pod="opendatahub/test-curl-8653-868444" Oct 30 17:13:16.275044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:16.275012 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8653-c4fb43"] Oct 30 17:13:16.285758 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:16.285715 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:16.287302 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:16.287272 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8653-868444\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="7eb2c034-c707-4550-81e3-cadb457fabcd" pod="opendatahub/test-curl-8653-868444" Oct 30 17:13:16.295459 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:13:16.295429 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b2c9fd9_f302_4d62_af98_a771e28d75d6.slice/crio-ff85b05863aba39d1839d5ff1cff97d3c4199d492ffe973ab657b9f80193d4ae WatchSource:0}: Error finding container ff85b05863aba39d1839d5ff1cff97d3c4199d492ffe973ab657b9f80193d4ae: Status 404 returned error can't find the container with id ff85b05863aba39d1839d5ff1cff97d3c4199d492ffe973ab657b9f80193d4ae Oct 30 17:13:16.303085 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:16.303043 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8653-868444\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="7eb2c034-c707-4550-81e3-cadb457fabcd" pod="opendatahub/test-curl-8653-868444" Oct 30 17:13:16.355317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:16.354110 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb"] Oct 30 17:13:16.360346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:16.360296 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-vsvmb"] Oct 30 17:13:16.839944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:16.839475 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" path="/var/lib/kubelet/pods/008aa481-f9ff-4742-b198-d8556de19b3a/volumes" Oct 30 17:13:16.973011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:16.956934 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:13:17.028600 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.028563 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-0aba40" Oct 30 17:13:17.055360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.054961 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-maas-api-tls\") pod \"2459de27-81ba-4d49-886e-8629d8cdbf60\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " Oct 30 17:13:17.058059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.058027 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-maas-api-tls" pod "2459de27-81ba-4d49-886e-8629d8cdbf60" (UID: "2459de27-81ba-4d49-886e-8629d8cdbf60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:17.119697 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.117260 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:17.156348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.156312 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4aa1cf3-5db3-4a4d-800d-e95249b49147-kube-api-access-2v88m\" (UniqueName: \"kubernetes.io/projected/e4aa1cf3-5db3-4a4d-800d-e95249b49147-kube-api-access-2v88m\") pod \"e4aa1cf3-5db3-4a4d-800d-e95249b49147\" (UID: \"e4aa1cf3-5db3-4a4d-800d-e95249b49147\") " Oct 30 17:13:17.156533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.156497 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-metrics-certs\") pod \"2459de27-81ba-4d49-886e-8629d8cdbf60\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " Oct 30 17:13:17.156650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.156632 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2459de27-81ba-4d49-886e-8629d8cdbf60-kube-api-access-2hscv\" (UniqueName: \"kubernetes.io/projected/2459de27-81ba-4d49-886e-8629d8cdbf60-kube-api-access-2hscv\") pod \"2459de27-81ba-4d49-886e-8629d8cdbf60\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " Oct 30 17:13:17.156786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.156770 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2459de27-81ba-4d49-886e-8629d8cdbf60-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2459de27-81ba-4d49-886e-8629d8cdbf60-openshift-service-ca\") pod \"2459de27-81ba-4d49-886e-8629d8cdbf60\" (UID: \"2459de27-81ba-4d49-886e-8629d8cdbf60\") " Oct 30 17:13:17.157106 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.157078 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2459de27-81ba-4d49-886e-8629d8cdbf60-openshift-service-ca" pod "2459de27-81ba-4d49-886e-8629d8cdbf60" (UID: "2459de27-81ba-4d49-886e-8629d8cdbf60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:17.157381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.157361 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2459de27-81ba-4d49-886e-8629d8cdbf60-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:17.157479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.157383 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:17.158788 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.158763 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2459de27-81ba-4d49-886e-8629d8cdbf60-kube-api-access-2hscv" pod "2459de27-81ba-4d49-886e-8629d8cdbf60" (UID: "2459de27-81ba-4d49-886e-8629d8cdbf60"). InnerVolumeSpecName "kube-api-access-2hscv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:17.159112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.159083 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-metrics-certs" pod "2459de27-81ba-4d49-886e-8629d8cdbf60" (UID: "2459de27-81ba-4d49-886e-8629d8cdbf60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:17.159112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.159086 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4aa1cf3-5db3-4a4d-800d-e95249b49147-kube-api-access-2v88m" pod "e4aa1cf3-5db3-4a4d-800d-e95249b49147" (UID: "e4aa1cf3-5db3-4a4d-800d-e95249b49147"). InnerVolumeSpecName "kube-api-access-2v88m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:17.254819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.254738 4828 generic.go:356] "Generic (PLEG): container finished" podID="e4aa1cf3-5db3-4a4d-800d-e95249b49147" containerID="2274afd804dea021044c66b68240000f8e0585e753988f64451a84e16ed8fa34" exitCode=137 Oct 30 17:13:17.254819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.254794 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-0aba40" Oct 30 17:13:17.255054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.254842 4828 scope.go:122] "RemoveContainer" containerID="2274afd804dea021044c66b68240000f8e0585e753988f64451a84e16ed8fa34" Oct 30 17:13:17.256408 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.256386 4828 generic.go:356] "Generic (PLEG): container finished" podID="2459de27-81ba-4d49-886e-8629d8cdbf60" containerID="5a80a93444f37aa6bad98f9f0dbe0b59e797cf81ba9ce1d4ad58a5ad27441a86" exitCode=0 Oct 30 17:13:17.256519 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.256457 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" Oct 30 17:13:17.256519 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.256478 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" event={"ID":"2459de27-81ba-4d49-886e-8629d8cdbf60","Type":"ContainerDied","Data":"5a80a93444f37aa6bad98f9f0dbe0b59e797cf81ba9ce1d4ad58a5ad27441a86"} Oct 30 17:13:17.256659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.256580 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g" event={"ID":"2459de27-81ba-4d49-886e-8629d8cdbf60","Type":"ContainerDied","Data":"3ba1622e437b1aee9edd841ec70fe00080e5a1a86100a6f97caf1ec839b88cb8"} Oct 30 17:13:17.261191 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.261159 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hscv\" (UniqueName: \"kubernetes.io/projected/2459de27-81ba-4d49-886e-8629d8cdbf60-kube-api-access-2hscv\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:17.261303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.261197 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v88m\" (UniqueName: \"kubernetes.io/projected/e4aa1cf3-5db3-4a4d-800d-e95249b49147-kube-api-access-2v88m\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:17.261303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.261216 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2459de27-81ba-4d49-886e-8629d8cdbf60-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:17.262478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.262458 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8653-c4fb43" event={"ID":"0b2c9fd9-f302-4d62-af98-a771e28d75d6","Type":"ContainerStarted","Data":"1c26f2212ec4c6c8ed45e8722804c88a48ee52760dbba2753d0c4cfdf1afadc9"} Oct 30 17:13:17.262573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.262492 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8653-c4fb43" event={"ID":"0b2c9fd9-f302-4d62-af98-a771e28d75d6","Type":"ContainerStarted","Data":"ff85b05863aba39d1839d5ff1cff97d3c4199d492ffe973ab657b9f80193d4ae"} Oct 30 17:13:17.273196 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.273148 4828 scope.go:122] "RemoveContainer" containerID="2274afd804dea021044c66b68240000f8e0585e753988f64451a84e16ed8fa34" Oct 30 17:13:17.273529 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:17.273506 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2274afd804dea021044c66b68240000f8e0585e753988f64451a84e16ed8fa34\": container with ID starting with 2274afd804dea021044c66b68240000f8e0585e753988f64451a84e16ed8fa34 not found: ID does not exist" containerID="2274afd804dea021044c66b68240000f8e0585e753988f64451a84e16ed8fa34" Oct 30 17:13:17.273621 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.273540 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2274afd804dea021044c66b68240000f8e0585e753988f64451a84e16ed8fa34"} err="failed to get container status \"2274afd804dea021044c66b68240000f8e0585e753988f64451a84e16ed8fa34\": rpc error: code = NotFound desc = could not find container \"2274afd804dea021044c66b68240000f8e0585e753988f64451a84e16ed8fa34\": container with ID starting with 2274afd804dea021044c66b68240000f8e0585e753988f64451a84e16ed8fa34 not found: ID does not exist" Oct 30 17:13:17.273621 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.273568 4828 scope.go:122] "RemoveContainer" containerID="5a80a93444f37aa6bad98f9f0dbe0b59e797cf81ba9ce1d4ad58a5ad27441a86" Oct 30 17:13:17.313941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.310879 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:17.327316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.326835 4828 scope.go:122] "RemoveContainer" containerID="5a80a93444f37aa6bad98f9f0dbe0b59e797cf81ba9ce1d4ad58a5ad27441a86" Oct 30 17:13:17.327316 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:17.327227 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a80a93444f37aa6bad98f9f0dbe0b59e797cf81ba9ce1d4ad58a5ad27441a86\": container with ID starting with 5a80a93444f37aa6bad98f9f0dbe0b59e797cf81ba9ce1d4ad58a5ad27441a86 not found: ID does not exist" containerID="5a80a93444f37aa6bad98f9f0dbe0b59e797cf81ba9ce1d4ad58a5ad27441a86" Oct 30 17:13:17.327316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.327261 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a80a93444f37aa6bad98f9f0dbe0b59e797cf81ba9ce1d4ad58a5ad27441a86"} err="failed to get container status \"5a80a93444f37aa6bad98f9f0dbe0b59e797cf81ba9ce1d4ad58a5ad27441a86\": rpc error: code = NotFound desc = could not find container \"5a80a93444f37aa6bad98f9f0dbe0b59e797cf81ba9ce1d4ad58a5ad27441a86\": container with ID starting with 5a80a93444f37aa6bad98f9f0dbe0b59e797cf81ba9ce1d4ad58a5ad27441a86 not found: ID does not exist" Oct 30 17:13:17.361181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.361109 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8653-c4fb43" podStartSLOduration=2.361091296 podStartE2EDuration="2.361091296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:17.360226701 +0000 UTC m=+1575.005736882" watchObservedRunningTime="2026-10-30 17:13:17.361091296 +0000 UTC m=+1575.006601477" Oct 30 17:13:17.403966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.403935 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g"] Oct 30 17:13:17.409172 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.409143 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-362bbf-fc96bbcf4-xlp7g"] Oct 30 17:13:17.415945 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:17.415876 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8653-0aba40\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="e4aa1cf3-5db3-4a4d-800d-e95249b49147" pod="opendatahub/test-curl-8653-0aba40" Oct 30 17:13:17.477050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.477014 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:17.787579 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:17.787539 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:18.298936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:18.298863 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8653-c4fb43"] Oct 30 17:13:18.310115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:18.310078 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8653-c4fb43"] Oct 30 17:13:18.495615 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:18.495577 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2459de27-81ba-4d49-886e-8629d8cdbf60" path="/var/lib/kubelet/pods/2459de27-81ba-4d49-886e-8629d8cdbf60/volumes" Oct 30 17:13:18.496348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:18.496324 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4aa1cf3-5db3-4a4d-800d-e95249b49147" path="/var/lib/kubelet/pods/e4aa1cf3-5db3-4a4d-800d-e95249b49147/volumes" Oct 30 17:13:18.501934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:18.501846 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-868444" Oct 30 17:13:18.584744 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:18.584702 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb2c034-c707-4550-81e3-cadb457fabcd-kube-api-access-mlv99\" (UniqueName: \"kubernetes.io/projected/7eb2c034-c707-4550-81e3-cadb457fabcd-kube-api-access-mlv99\") pod \"7eb2c034-c707-4550-81e3-cadb457fabcd\" (UID: \"7eb2c034-c707-4550-81e3-cadb457fabcd\") " Oct 30 17:13:18.586955 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:18.586918 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb2c034-c707-4550-81e3-cadb457fabcd-kube-api-access-mlv99" pod "7eb2c034-c707-4550-81e3-cadb457fabcd" (UID: "7eb2c034-c707-4550-81e3-cadb457fabcd"). InnerVolumeSpecName "kube-api-access-mlv99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:18.686657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:18.686613 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlv99\" (UniqueName: \"kubernetes.io/projected/7eb2c034-c707-4550-81e3-cadb457fabcd-kube-api-access-mlv99\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:18.965524 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:18.965481 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:19.286351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.286313 4828 generic.go:356] "Generic (PLEG): container finished" podID="7eb2c034-c707-4550-81e3-cadb457fabcd" containerID="63bf837f1e477e4895f59e69fa45135a7925a9843b5d1c2b64ebbfb6570b9940" exitCode=137 Oct 30 17:13:19.286565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.286540 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8653-c4fb43" podUID="0b2c9fd9-f302-4d62-af98-a771e28d75d6" containerName="test-curl-8653-c4fb43" containerID="cri-o://1c26f2212ec4c6c8ed45e8722804c88a48ee52760dbba2753d0c4cfdf1afadc9" gracePeriod=2 Oct 30 17:13:19.286850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.286823 4828 scope.go:122] "RemoveContainer" containerID="63bf837f1e477e4895f59e69fa45135a7925a9843b5d1c2b64ebbfb6570b9940" Oct 30 17:13:19.287038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.286956 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-868444" Oct 30 17:13:19.301394 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.301360 4828 scope.go:122] "RemoveContainer" containerID="63bf837f1e477e4895f59e69fa45135a7925a9843b5d1c2b64ebbfb6570b9940" Oct 30 17:13:19.301696 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:19.301682 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63bf837f1e477e4895f59e69fa45135a7925a9843b5d1c2b64ebbfb6570b9940\": container with ID starting with 63bf837f1e477e4895f59e69fa45135a7925a9843b5d1c2b64ebbfb6570b9940 not found: ID does not exist" containerID="63bf837f1e477e4895f59e69fa45135a7925a9843b5d1c2b64ebbfb6570b9940" Oct 30 17:13:19.301755 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.301709 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63bf837f1e477e4895f59e69fa45135a7925a9843b5d1c2b64ebbfb6570b9940"} err="failed to get container status \"63bf837f1e477e4895f59e69fa45135a7925a9843b5d1c2b64ebbfb6570b9940\": rpc error: code = NotFound desc = could not find container \"63bf837f1e477e4895f59e69fa45135a7925a9843b5d1c2b64ebbfb6570b9940\": container with ID starting with 63bf837f1e477e4895f59e69fa45135a7925a9843b5d1c2b64ebbfb6570b9940 not found: ID does not exist" Oct 30 17:13:19.315733 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:19.315698 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8653-868444\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="7eb2c034-c707-4550-81e3-cadb457fabcd" pod="opendatahub/test-curl-8653-868444" Oct 30 17:13:19.413390 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.413350 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:19.432232 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.432194 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd"] Oct 30 17:13:19.433060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.433040 4828 cpu_manager.go:414] "containerMap: removing container" podUID="0b2c9fd9-f302-4d62-af98-a771e28d75d6" containerName="test-curl-8653-c4fb43" Oct 30 17:13:19.433060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.433057 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b2c9fd9-f302-4d62-af98-a771e28d75d6" containerName="test-curl-8653-c4fb43" Oct 30 17:13:19.433259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.433074 4828 cpu_manager.go:414] "containerMap: removing container" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" containerName="istio-proxy" Oct 30 17:13:19.433259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.433079 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" containerName="istio-proxy" Oct 30 17:13:19.433259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.433104 4828 cpu_manager.go:414] "containerMap: removing container" podUID="2459de27-81ba-4d49-886e-8629d8cdbf60" containerName="maas-api" Oct 30 17:13:19.433259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.433109 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2459de27-81ba-4d49-886e-8629d8cdbf60" containerName="maas-api" Oct 30 17:13:19.433259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.433236 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b2c9fd9-f302-4d62-af98-a771e28d75d6" containerName="test-curl-8653-c4fb43" Oct 30 17:13:19.433259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.433246 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="008aa481-f9ff-4742-b198-d8556de19b3a" containerName="istio-proxy" Oct 30 17:13:19.433259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.433255 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="2459de27-81ba-4d49-886e-8629d8cdbf60" containerName="maas-api" Oct 30 17:13:19.437044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.437021 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.451313 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:19.451289 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8653-868444\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="7eb2c034-c707-4550-81e3-cadb457fabcd" pod="opendatahub/test-curl-8653-868444" Oct 30 17:13:19.458994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.458973 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d003a5-openshift-default-dockercfg-8dqnq\"" Oct 30 17:13:19.464950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.464924 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd"] Oct 30 17:13:19.500548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.500508 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-token\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.500722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.500692 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-credential-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.500820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.500729 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-data\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.500820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.500762 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istiod-ca-cert\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.500962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.500941 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-podinfo\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.501018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.500994 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.501073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.501050 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-envoy\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.501130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.501083 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7cpr\" (UniqueName: \"kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-kube-api-access-c7cpr\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.501305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.501274 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-certs\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.583371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.583251 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:19.603740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.603541 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-podinfo\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.603740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.603636 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.603740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.603708 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-envoy\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.606617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.603750 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7cpr\" (UniqueName: \"kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-kube-api-access-c7cpr\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.606617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.603817 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-certs\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.606617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.603990 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-token\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.606617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.604048 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-credential-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.606617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.604080 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-data\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.606617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.604160 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istiod-ca-cert\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.621938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.619341 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istiod-ca-cert\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.637319 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.637248 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-certs\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.642283 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.641211 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-data\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.642817 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.642777 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-credential-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.676612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.676576 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-socket\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.686956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.678628 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-podinfo\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.696941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.696833 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-token\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.709938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.701982 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-envoy\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.710998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.710930 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7cpr\" (UniqueName: \"kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-kube-api-access-c7cpr\") pod \"e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.761050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.758340 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:19.775113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:19.775063 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:20.222269 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:20.222219 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd"] Oct 30 17:13:20.222874 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:13:20.222850 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d19b8e4_861b_4a5a_bb93_821c92e3a5e4.slice/crio-5b2815f214e4be548180059411516fdbda65caf48613aa07a46af3fe32e9fe39 WatchSource:0}: Error finding container 5b2815f214e4be548180059411516fdbda65caf48613aa07a46af3fe32e9fe39: Status 404 returned error can't find the container with id 5b2815f214e4be548180059411516fdbda65caf48613aa07a46af3fe32e9fe39 Oct 30 17:13:20.225686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:20.225650 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:20.225812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:20.225735 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:20.225812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:20.225781 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:20.302950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:20.302876 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" event={"ID":"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4","Type":"ContainerStarted","Data":"5b2815f214e4be548180059411516fdbda65caf48613aa07a46af3fe32e9fe39"} Oct 30 17:13:20.514909 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:20.514849 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eb2c034-c707-4550-81e3-cadb457fabcd" path="/var/lib/kubelet/pods/7eb2c034-c707-4550-81e3-cadb457fabcd/volumes" Oct 30 17:13:21.314739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:21.314706 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" event={"ID":"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4","Type":"ContainerStarted","Data":"3921be05aa428d4d55d5f0e1dbc05c2cbfea1131975e2a2d97e29f784c0b47cc"} Oct 30 17:13:21.390958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:21.390868 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" podStartSLOduration=2.390847301 podStartE2EDuration="2.390847301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:21.374705449 +0000 UTC m=+1579.020215632" watchObservedRunningTime="2026-10-30 17:13:21.390847301 +0000 UTC m=+1579.036357482" Oct 30 17:13:21.460456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:21.460413 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:21.605924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:21.604145 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-c4fb43" Oct 30 17:13:21.663402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:21.663360 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2c9fd9-f302-4d62-af98-a771e28d75d6-kube-api-access-shtgk\" (UniqueName: \"kubernetes.io/projected/0b2c9fd9-f302-4d62-af98-a771e28d75d6-kube-api-access-shtgk\") pod \"0b2c9fd9-f302-4d62-af98-a771e28d75d6\" (UID: \"0b2c9fd9-f302-4d62-af98-a771e28d75d6\") " Oct 30 17:13:21.665437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:21.665411 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2c9fd9-f302-4d62-af98-a771e28d75d6-kube-api-access-shtgk" pod "0b2c9fd9-f302-4d62-af98-a771e28d75d6" (UID: "0b2c9fd9-f302-4d62-af98-a771e28d75d6"). InnerVolumeSpecName "kube-api-access-shtgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:21.759426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:21.759391 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:21.766208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:21.766178 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:21.766376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:21.766352 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shtgk\" (UniqueName: \"kubernetes.io/projected/0b2c9fd9-f302-4d62-af98-a771e28d75d6-kube-api-access-shtgk\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:22.323404 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:22.323364 4828 generic.go:356] "Generic (PLEG): container finished" podID="0b2c9fd9-f302-4d62-af98-a771e28d75d6" containerID="1c26f2212ec4c6c8ed45e8722804c88a48ee52760dbba2753d0c4cfdf1afadc9" exitCode=137 Oct 30 17:13:22.323877 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:22.323412 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8653-c4fb43" Oct 30 17:13:22.323877 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:22.323430 4828 scope.go:122] "RemoveContainer" containerID="1c26f2212ec4c6c8ed45e8722804c88a48ee52760dbba2753d0c4cfdf1afadc9" Oct 30 17:13:22.324049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:22.323921 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:22.325032 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:22.325014 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:13:22.337357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:22.337330 4828 scope.go:122] "RemoveContainer" containerID="1c26f2212ec4c6c8ed45e8722804c88a48ee52760dbba2753d0c4cfdf1afadc9" Oct 30 17:13:22.358110 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:22.337980 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c26f2212ec4c6c8ed45e8722804c88a48ee52760dbba2753d0c4cfdf1afadc9\": container with ID starting with 1c26f2212ec4c6c8ed45e8722804c88a48ee52760dbba2753d0c4cfdf1afadc9 not found: ID does not exist" containerID="1c26f2212ec4c6c8ed45e8722804c88a48ee52760dbba2753d0c4cfdf1afadc9" Oct 30 17:13:22.358110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:22.338210 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c26f2212ec4c6c8ed45e8722804c88a48ee52760dbba2753d0c4cfdf1afadc9"} err="failed to get container status \"1c26f2212ec4c6c8ed45e8722804c88a48ee52760dbba2753d0c4cfdf1afadc9\": rpc error: code = NotFound desc = could not find container \"1c26f2212ec4c6c8ed45e8722804c88a48ee52760dbba2753d0c4cfdf1afadc9\": container with ID starting with 1c26f2212ec4c6c8ed45e8722804c88a48ee52760dbba2753d0c4cfdf1afadc9 not found: ID does not exist" Oct 30 17:13:22.447964 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:22.441147 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8653-c4fb43\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="0b2c9fd9-f302-4d62-af98-a771e28d75d6" pod="opendatahub/test-curl-8653-c4fb43" Oct 30 17:13:22.545927 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:22.542306 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8653-c4fb43\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="0b2c9fd9-f302-4d62-af98-a771e28d75d6" pod="opendatahub/test-curl-8653-c4fb43" Oct 30 17:13:22.676044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:22.568687 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b2c9fd9-f302-4d62-af98-a771e28d75d6" path="/var/lib/kubelet/pods/0b2c9fd9-f302-4d62-af98-a771e28d75d6/volumes" Oct 30 17:13:24.170063 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:24.166984 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:13:24.242060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:24.234008 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.145:8000/health\": context deadline exceeded" Oct 30 17:13:24.710972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:24.710832 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:24.753925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:24.753761 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:13:24.768092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:24.768032 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:13:24.800192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:24.800133 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:13:24.892922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:24.892701 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:24.922035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:24.921986 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.145:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:13:25.074819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:25.074778 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:25.706926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:25.687179 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:25.841679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:25.841618 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": context deadline exceeded" Oct 30 17:13:25.869935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:25.852928 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:13:26.084926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:25.856016 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:13:26.606160 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:26.064590 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:13:26.606449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:26.064747 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.145:8000/ready\": context deadline exceeded" Oct 30 17:13:26.606587 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:26.287318 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:13:26.611995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:26.533446 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:26.638934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:26.638313 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:13:26.699606 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:26.699557 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:13:26.855734 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:26.855690 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": context deadline exceeded" Oct 30 17:13:26.988196 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:26.988024 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:13:27.095929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.071083 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:13:27.167927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.118944 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:13:27.206933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.174063 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:13:27.537100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.536834 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:27.578311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.578271 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:13:27.578578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.578562 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98"] Oct 30 17:13:27.752666 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.752542 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:27.813156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.813081 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:13:27.813379 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.813363 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98"] Oct 30 17:13:27.813516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.813505 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:13:27.813654 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.813643 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:13:27.813785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.813774 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs"] Oct 30 17:13:27.814586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.814430 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:27.822116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.822089 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs"] Oct 30 17:13:27.822372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.822354 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" Oct 30 17:13:27.838483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.838460 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d003a5\"" Oct 30 17:13:27.840255 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.840023 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f42baf-openshift-default-dockercfg-njzvm\"" Oct 30 17:13:27.842582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.840078 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:13:27.842582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.840212 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d003a5-dockercfg-5blbx\"" Oct 30 17:13:27.910923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.910609 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-istio-token\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:27.910923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.910691 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-envoy\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:27.910923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.910758 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-credential-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:27.910923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.910785 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-certs\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:27.910923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.910823 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300e1607-7771-4378-b40a-56cf985f25d5-istio-podinfo\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:27.910923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.910861 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:27.911680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.910960 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcbjk\" (UniqueName: \"kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-kube-api-access-mcbjk\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:27.911680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.910996 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-data\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:27.911680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.911088 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxvvk\" (UniqueName: \"kubernetes.io/projected/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-kube-api-access-vxvvk\") pod \"payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs\" (UID: \"ea897670-fd5e-45cc-8529-7cbe6a7d6a06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" Oct 30 17:13:27.911680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.911127 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs\" (UID: \"ea897670-fd5e-45cc-8529-7cbe6a7d6a06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" Oct 30 17:13:27.911680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:27.911183 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300e1607-7771-4378-b40a-56cf985f25d5-istiod-ca-cert\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.014926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.014446 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-credential-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.014926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.014497 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-certs\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.014926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.014537 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300e1607-7771-4378-b40a-56cf985f25d5-istio-podinfo\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.014926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.014570 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.014926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.014640 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcbjk\" (UniqueName: \"kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-kube-api-access-mcbjk\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.014926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.014672 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-data\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.014926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.014757 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxvvk\" (UniqueName: \"kubernetes.io/projected/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-kube-api-access-vxvvk\") pod \"payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs\" (UID: \"ea897670-fd5e-45cc-8529-7cbe6a7d6a06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" Oct 30 17:13:28.014926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.014796 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs\" (UID: \"ea897670-fd5e-45cc-8529-7cbe6a7d6a06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" Oct 30 17:13:28.014926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.014848 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300e1607-7771-4378-b40a-56cf985f25d5-istiod-ca-cert\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.021595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.014978 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-istio-token\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.021595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.015045 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-envoy\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.021595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.016158 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-credential-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.021595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.016435 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-certs\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.021595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.017497 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300e1607-7771-4378-b40a-56cf985f25d5-istiod-ca-cert\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.021595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.018019 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-data\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.043969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.032271 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.043969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.038009 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs\" (UID: \"ea897670-fd5e-45cc-8529-7cbe6a7d6a06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" Oct 30 17:13:28.059992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.051798 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcbjk\" (UniqueName: \"kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-kube-api-access-mcbjk\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.076976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.076263 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-envoy\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.384135 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.374146 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300e1607-7771-4378-b40a-56cf985f25d5-istio-podinfo\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.384135 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.374792 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-istio-token\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.384135 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.375477 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxvvk\" (UniqueName: \"kubernetes.io/projected/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-kube-api-access-vxvvk\") pod \"payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs\" (UID: \"ea897670-fd5e-45cc-8529-7cbe6a7d6a06\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" Oct 30 17:13:28.384135 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.376865 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" Oct 30 17:13:28.400200 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.393392 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:28.953972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.953925 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:28.954310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.954033 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:28.954310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.954091 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:28.954644 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.954624 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98"] Oct 30 17:13:28.961970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:28.961945 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs"] Oct 30 17:13:29.237288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:29.237229 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:29.527925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:29.512769 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" event={"ID":"ea897670-fd5e-45cc-8529-7cbe6a7d6a06","Type":"ContainerStarted","Data":"9e04df38ba8f5c1861084ea111df06772027576f488e41b40c03cfb2088ca0bb"} Oct 30 17:13:29.527925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:29.519058 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" event={"ID":"ea897670-fd5e-45cc-8529-7cbe6a7d6a06","Type":"ContainerStarted","Data":"5a4ddc7d6f8a9c9a3a669935ed7e15e998d6a1a286859d81d666939b2dfc37b8"} Oct 30 17:13:29.527925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:29.521342 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" Oct 30 17:13:29.569585 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:29.569547 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" event={"ID":"300e1607-7771-4378-b40a-56cf985f25d5","Type":"ContainerStarted","Data":"6ab43b9e3392c1afa54e24d89e2cdf67e66ade2420a9e7c8bf9588f3e481fd95"} Oct 30 17:13:29.570967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:29.570937 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" event={"ID":"300e1607-7771-4378-b40a-56cf985f25d5","Type":"ContainerStarted","Data":"7669a00dafec1a23e1028fa176a4181f33ea5c87c485a78d2bfbf131b635d625"} Oct 30 17:13:29.580240 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:29.580170 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" podStartSLOduration=2.580150809 podStartE2EDuration="2.580150809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:29.578613334 +0000 UTC m=+1587.224123515" watchObservedRunningTime="2026-10-30 17:13:29.580150809 +0000 UTC m=+1587.225660991" Oct 30 17:13:29.625548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:29.613242 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:29.634078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:29.634026 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" podStartSLOduration=3.634012472 podStartE2EDuration="3.634012472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:29.627653564 +0000 UTC m=+1587.273163747" watchObservedRunningTime="2026-10-30 17:13:29.634012472 +0000 UTC m=+1587.279522652" Oct 30 17:13:30.409396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:30.409217 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:30.415924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:30.415877 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:30.592805 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:30.592768 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:30.594001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:30.593977 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:13:30.867756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:30.867720 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:31.456924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:31.456802 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:32.005126 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:32.005075 4828 patch_prober.go:29] interesting pod/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:13:32.005341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:32.005151 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:13:32.123640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:32.123595 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:32.327362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:32.327328 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:33.537428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:33.537386 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:33.854624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:33.854538 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:34.002638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:34.002603 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:35.290457 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:35.290372 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:35.674701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:35.674602 4828 generic.go:356] "Generic (PLEG): container finished" podID="ba9d4488-f73e-4096-bed3-603700716223" containerID="16027ede312e132adb65d31837091d70f90ed6e202d897927844852178931959" exitCode=0 Oct 30 17:13:35.674876 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:35.674701 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" event={"ID":"ba9d4488-f73e-4096-bed3-603700716223","Type":"ContainerDied","Data":"16027ede312e132adb65d31837091d70f90ed6e202d897927844852178931959"} Oct 30 17:13:35.675312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:35.675292 4828 scope.go:122] "RemoveContainer" containerID="16027ede312e132adb65d31837091d70f90ed6e202d897927844852178931959" Oct 30 17:13:35.678745 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:35.678700 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:35.678948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:35.678885 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:35.679055 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:35.679035 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:13:36.717653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:36.717614 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" event={"ID":"ba9d4488-f73e-4096-bed3-603700716223","Type":"ContainerStarted","Data":"0248b2bec6d1bae209786cc66e392611540a6ed1b4200de12c5b35bab71b5188"} Oct 30 17:13:37.061645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:37.061609 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:37.266358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:37.266321 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:13:37.266564 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:37.266542 4828 patch_prober.go:29] interesting pod/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 17:13:37.266650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:37.266613 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 17:13:37.932680 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:37.932635 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:13:38.266298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:38.266265 4828 patch_prober.go:29] interesting pod/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 17:13:38.266470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:38.266338 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 17:13:38.814604 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:38.814463 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:39.266289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:39.266249 4828 patch_prober.go:29] interesting pod/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 17:13:39.266734 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:39.266334 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 17:13:40.266497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:40.266461 4828 patch_prober.go:29] interesting pod/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 17:13:40.266916 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:40.266527 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 17:13:40.607717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:40.607581 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:41.265990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:41.265949 4828 patch_prober.go:29] interesting pod/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 17:13:41.266178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:41.266023 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 17:13:41.606605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:41.606520 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" Oct 30 17:13:41.814759 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:41.814718 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr"] Oct 30 17:13:41.854472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:41.854439 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk"] Oct 30 17:13:42.023646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:42.023604 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:42.182967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:42.172244 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:42.266386 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:42.266345 4828 patch_prober.go:29] interesting pod/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 17:13:42.266568 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:42.266414 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 17:13:42.492588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:42.492423 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:43.271067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:43.271033 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:13:43.271575 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:43.271555 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:13:43.272263 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:43.272241 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:13:43.993923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:43.991770 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:44.452825 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:44.451646 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:44.951671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:44.951633 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:46.252267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:46.252222 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:46.815730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:46.815675 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" podUID="a8eb2ac7-e218-4729-bc72-578ee763e55a" containerName="payload-processing" containerID="cri-o://8a167aba21297860709b3717138489328ac6d0f525947258eae4b8cfc012fad7" gracePeriod=25 Oct 30 17:13:46.855157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:46.855112 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" podUID="53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a" containerName="payload-pre-processing" containerID="cri-o://e2843c27811da4f379b7c02119bd436dac1028fa2f5bc7bb10207fdab54e2e97" gracePeriod=25 Oct 30 17:13:47.179804 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.179771 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" Oct 30 17:13:47.210702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.210673 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" Oct 30 17:13:47.287516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.287487 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-kube-api-access-6v8mv\" (UniqueName: \"kubernetes.io/projected/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-kube-api-access-6v8mv\") pod \"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a\" (UID: \"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a\") " Oct 30 17:13:47.288024 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.287564 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8eb2ac7-e218-4729-bc72-578ee763e55a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8eb2ac7-e218-4729-bc72-578ee763e55a-plugins-config-volume\") pod \"a8eb2ac7-e218-4729-bc72-578ee763e55a\" (UID: \"a8eb2ac7-e218-4729-bc72-578ee763e55a\") " Oct 30 17:13:47.288024 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.287687 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8eb2ac7-e218-4729-bc72-578ee763e55a-kube-api-access-z798x\" (UniqueName: \"kubernetes.io/projected/a8eb2ac7-e218-4729-bc72-578ee763e55a-kube-api-access-z798x\") pod \"a8eb2ac7-e218-4729-bc72-578ee763e55a\" (UID: \"a8eb2ac7-e218-4729-bc72-578ee763e55a\") " Oct 30 17:13:47.288024 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.287728 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-plugins-config-volume\") pod \"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a\" (UID: \"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a\") " Oct 30 17:13:47.288646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.288621 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-plugins-config-volume" pod "53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a" (UID: "53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:47.288816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.288627 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8eb2ac7-e218-4729-bc72-578ee763e55a-plugins-config-volume" pod "a8eb2ac7-e218-4729-bc72-578ee763e55a" (UID: "a8eb2ac7-e218-4729-bc72-578ee763e55a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:47.291042 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.290967 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-kube-api-access-6v8mv" pod "53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a" (UID: "53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a"). InnerVolumeSpecName "kube-api-access-6v8mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:47.291042 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.291021 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8eb2ac7-e218-4729-bc72-578ee763e55a-kube-api-access-z798x" pod "a8eb2ac7-e218-4729-bc72-578ee763e55a" (UID: "a8eb2ac7-e218-4729-bc72-578ee763e55a"). InnerVolumeSpecName "kube-api-access-z798x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:47.389933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.389836 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8eb2ac7-e218-4729-bc72-578ee763e55a-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.389933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.389877 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z798x\" (UniqueName: \"kubernetes.io/projected/a8eb2ac7-e218-4729-bc72-578ee763e55a-kube-api-access-z798x\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.389933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.389894 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.389933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.389924 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v8mv\" (UniqueName: \"kubernetes.io/projected/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a-kube-api-access-6v8mv\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.931515 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.931477 4828 generic.go:356] "Generic (PLEG): container finished" podID="a8eb2ac7-e218-4729-bc72-578ee763e55a" containerID="8a167aba21297860709b3717138489328ac6d0f525947258eae4b8cfc012fad7" exitCode=0 Oct 30 17:13:47.931710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.931531 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" Oct 30 17:13:47.931710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.931530 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" event={"ID":"a8eb2ac7-e218-4729-bc72-578ee763e55a","Type":"ContainerDied","Data":"8a167aba21297860709b3717138489328ac6d0f525947258eae4b8cfc012fad7"} Oct 30 17:13:47.931710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.931597 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr" event={"ID":"a8eb2ac7-e218-4729-bc72-578ee763e55a","Type":"ContainerDied","Data":"81cceea6e5d9d608080271a437ca37aa861bcec94576fddfea2206368bdcb94c"} Oct 30 17:13:47.931710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.931627 4828 scope.go:122] "RemoveContainer" containerID="8a167aba21297860709b3717138489328ac6d0f525947258eae4b8cfc012fad7" Oct 30 17:13:47.934191 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.934169 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:47.935558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.935529 4828 generic.go:356] "Generic (PLEG): container finished" podID="53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a" containerID="e2843c27811da4f379b7c02119bd436dac1028fa2f5bc7bb10207fdab54e2e97" exitCode=0 Oct 30 17:13:47.935690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.935572 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" Oct 30 17:13:47.935690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.935575 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" event={"ID":"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a","Type":"ContainerDied","Data":"e2843c27811da4f379b7c02119bd436dac1028fa2f5bc7bb10207fdab54e2e97"} Oct 30 17:13:47.935690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.935625 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk" event={"ID":"53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a","Type":"ContainerDied","Data":"7ace575053b271d2ca222c6a01a8255e7ce9c917dd7b2225829068d7f38b93fc"} Oct 30 17:13:47.945509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.945491 4828 scope.go:122] "RemoveContainer" containerID="8a167aba21297860709b3717138489328ac6d0f525947258eae4b8cfc012fad7" Oct 30 17:13:47.945875 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:47.945854 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a167aba21297860709b3717138489328ac6d0f525947258eae4b8cfc012fad7\": container with ID starting with 8a167aba21297860709b3717138489328ac6d0f525947258eae4b8cfc012fad7 not found: ID does not exist" containerID="8a167aba21297860709b3717138489328ac6d0f525947258eae4b8cfc012fad7" Oct 30 17:13:47.946011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.945881 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a167aba21297860709b3717138489328ac6d0f525947258eae4b8cfc012fad7"} err="failed to get container status \"8a167aba21297860709b3717138489328ac6d0f525947258eae4b8cfc012fad7\": rpc error: code = NotFound desc = could not find container \"8a167aba21297860709b3717138489328ac6d0f525947258eae4b8cfc012fad7\": container with ID starting with 8a167aba21297860709b3717138489328ac6d0f525947258eae4b8cfc012fad7 not found: ID does not exist" Oct 30 17:13:47.946011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.945920 4828 scope.go:122] "RemoveContainer" containerID="e2843c27811da4f379b7c02119bd436dac1028fa2f5bc7bb10207fdab54e2e97" Oct 30 17:13:47.957152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.957134 4828 scope.go:122] "RemoveContainer" containerID="e2843c27811da4f379b7c02119bd436dac1028fa2f5bc7bb10207fdab54e2e97" Oct 30 17:13:47.957461 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:13:47.957440 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2843c27811da4f379b7c02119bd436dac1028fa2f5bc7bb10207fdab54e2e97\": container with ID starting with e2843c27811da4f379b7c02119bd436dac1028fa2f5bc7bb10207fdab54e2e97 not found: ID does not exist" containerID="e2843c27811da4f379b7c02119bd436dac1028fa2f5bc7bb10207fdab54e2e97" Oct 30 17:13:47.957526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.957470 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2843c27811da4f379b7c02119bd436dac1028fa2f5bc7bb10207fdab54e2e97"} err="failed to get container status \"e2843c27811da4f379b7c02119bd436dac1028fa2f5bc7bb10207fdab54e2e97\": rpc error: code = NotFound desc = could not find container \"e2843c27811da4f379b7c02119bd436dac1028fa2f5bc7bb10207fdab54e2e97\": container with ID starting with e2843c27811da4f379b7c02119bd436dac1028fa2f5bc7bb10207fdab54e2e97 not found: ID does not exist" Oct 30 17:13:47.992138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:47.992108 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr"] Oct 30 17:13:48.000431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:48.000377 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-432680-85c94bccd8-gpspr"] Oct 30 17:13:48.097045 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:48.096958 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk"] Oct 30 17:13:48.101816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:48.101520 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-432680-8bcb57d95-bhbfk"] Oct 30 17:13:48.497153 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:48.497075 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a" path="/var/lib/kubelet/pods/53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a/volumes" Oct 30 17:13:48.497708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:48.497687 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8eb2ac7-e218-4729-bc72-578ee763e55a" path="/var/lib/kubelet/pods/a8eb2ac7-e218-4729-bc72-578ee763e55a/volumes" Oct 30 17:13:49.633925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:49.622589 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:51.734769 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:51.734675 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:52.490064 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:52.490025 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:52.653114 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:52.653066 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:52.814011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:52.813976 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44"] Oct 30 17:13:52.814806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:52.814764 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" containerID="cri-o://c8b4a75c8b026693d7ea4d1015f4fb19e8cccbe8a314552e3d772411ac37c899" gracePeriod=30 Oct 30 17:13:52.873857 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:52.873820 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:53.539601 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:53.539558 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:53.607375 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:53.607339 4828 patch_prober.go:29] interesting pod/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:13:53.607569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:53.607540 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:13:55.240930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:55.240524 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:56.528560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:56.528518 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:56.929518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:56.929438 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:58.077822 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.077792 4828 generic.go:356] "Generic (PLEG): container finished" podID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerID="c8b4a75c8b026693d7ea4d1015f4fb19e8cccbe8a314552e3d772411ac37c899" exitCode=0 Oct 30 17:13:58.078295 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.077862 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" event={"ID":"836c4dab-6052-49e4-b60e-98bfa0f48b5b","Type":"ContainerDied","Data":"c8b4a75c8b026693d7ea4d1015f4fb19e8cccbe8a314552e3d772411ac37c899"} Oct 30 17:13:58.078295 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.077941 4828 scope.go:122] "RemoveContainer" containerID="e2ca8adfbdccf2e68d612c0af1c36e56d7770ab912d1d8bcc4e57af7ea3b65ad" Oct 30 17:13:58.113710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.113685 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:13:58.230053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.230016 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-credential-socket\") pod \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " Oct 30 17:13:58.230229 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.230199 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-kube-api-access-rvcb9\" (UniqueName: \"kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-kube-api-access-rvcb9\") pod \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " Oct 30 17:13:58.230284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.230272 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-podinfo\") pod \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " Oct 30 17:13:58.230323 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.230307 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istiod-ca-cert\") pod \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " Oct 30 17:13:58.230366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.230345 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-certs\") pod \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " Oct 30 17:13:58.230458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.230385 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-token\" (UniqueName: \"kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-token\") pod \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " Oct 30 17:13:58.230458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.230414 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-socket\") pod \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " Oct 30 17:13:58.230544 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.230480 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-envoy\") pod \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " Oct 30 17:13:58.230544 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.230520 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-credential-socket" pod "836c4dab-6052-49e4-b60e-98bfa0f48b5b" (UID: "836c4dab-6052-49e4-b60e-98bfa0f48b5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:58.230624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.230560 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-data\") pod \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\" (UID: \"836c4dab-6052-49e4-b60e-98bfa0f48b5b\") " Oct 30 17:13:58.230796 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.230775 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-certs" pod "836c4dab-6052-49e4-b60e-98bfa0f48b5b" (UID: "836c4dab-6052-49e4-b60e-98bfa0f48b5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:58.231046 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.231025 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-data" pod "836c4dab-6052-49e4-b60e-98bfa0f48b5b" (UID: "836c4dab-6052-49e4-b60e-98bfa0f48b5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:58.231348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.231329 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.231455 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.231350 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.231455 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.231364 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.232053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.232024 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-socket" pod "836c4dab-6052-49e4-b60e-98bfa0f48b5b" (UID: "836c4dab-6052-49e4-b60e-98bfa0f48b5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:58.232204 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.232184 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istiod-ca-cert" pod "836c4dab-6052-49e4-b60e-98bfa0f48b5b" (UID: "836c4dab-6052-49e4-b60e-98bfa0f48b5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:58.234115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.234078 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-podinfo" pod "836c4dab-6052-49e4-b60e-98bfa0f48b5b" (UID: "836c4dab-6052-49e4-b60e-98bfa0f48b5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:13:58.234211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.234111 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-token" pod "836c4dab-6052-49e4-b60e-98bfa0f48b5b" (UID: "836c4dab-6052-49e4-b60e-98bfa0f48b5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:58.234449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.234426 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-envoy" pod "836c4dab-6052-49e4-b60e-98bfa0f48b5b" (UID: "836c4dab-6052-49e4-b60e-98bfa0f48b5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:58.236463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.236442 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-kube-api-access-rvcb9" pod "836c4dab-6052-49e4-b60e-98bfa0f48b5b" (UID: "836c4dab-6052-49e4-b60e-98bfa0f48b5b"). InnerVolumeSpecName "kube-api-access-rvcb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:58.332953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.332826 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvcb9\" (UniqueName: \"kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-kube-api-access-rvcb9\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.332953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.332856 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.332953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.332866 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.332953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.332875 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.332953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.332883 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.332953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.332892 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/836c4dab-6052-49e4-b60e-98bfa0f48b5b-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.603691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.598100 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt"] Oct 30 17:13:58.603691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.599293 4828 cpu_manager.go:414] "containerMap: removing container" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" Oct 30 17:13:58.603691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.599315 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" Oct 30 17:13:58.603691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.599352 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a8eb2ac7-e218-4729-bc72-578ee763e55a" containerName="payload-processing" Oct 30 17:13:58.603691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.599360 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8eb2ac7-e218-4729-bc72-578ee763e55a" containerName="payload-processing" Oct 30 17:13:58.603691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.599381 4828 cpu_manager.go:414] "containerMap: removing container" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" Oct 30 17:13:58.603691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.599389 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" Oct 30 17:13:58.603691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.599415 4828 cpu_manager.go:414] "containerMap: removing container" podUID="53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a" containerName="payload-pre-processing" Oct 30 17:13:58.603691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.599423 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a" containerName="payload-pre-processing" Oct 30 17:13:58.603691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.599626 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" Oct 30 17:13:58.603691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.599642 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8eb2ac7-e218-4729-bc72-578ee763e55a" containerName="payload-processing" Oct 30 17:13:58.603691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.599658 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b3dd0f-6ae2-42e7-ad7c-2c7eb187c62a" containerName="payload-pre-processing" Oct 30 17:13:58.604445 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.604184 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:58.622674 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.622640 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d003a5-serving-cert\"" Oct 30 17:13:58.630555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.629942 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt"] Oct 30 17:13:58.637020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.636984 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwsv\" (UniqueName: \"kubernetes.io/projected/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-kube-api-access-frwsv\") pod \"maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:58.637169 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.637034 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:58.637243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.637222 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-metrics-certs\") pod \"maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:58.637348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.637328 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-maas-api-tls\") pod \"maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:58.691878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.691837 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:58.739151 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.739120 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwsv\" (UniqueName: \"kubernetes.io/projected/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-kube-api-access-frwsv\") pod \"maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:58.739336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.739155 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:58.739336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.739224 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-metrics-certs\") pod \"maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:58.739336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.739273 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-maas-api-tls\") pod \"maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:58.739998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.739969 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:58.741685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.741663 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-maas-api-tls\") pod \"maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:58.741783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.741769 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-metrics-certs\") pod \"maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:58.763065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.763038 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwsv\" (UniqueName: \"kubernetes.io/projected/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-kube-api-access-frwsv\") pod \"maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:58.885982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.885628 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:13:58.942980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:58.936963 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:13:59.099246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:59.091064 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" event={"ID":"836c4dab-6052-49e4-b60e-98bfa0f48b5b","Type":"ContainerDied","Data":"70d48691a9fb20e27525b0b5fd33ec8cd53aad5852e8f47313a5fc313f04648a"} Oct 30 17:13:59.099246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:59.091239 4828 scope.go:122] "RemoveContainer" containerID="c8b4a75c8b026693d7ea4d1015f4fb19e8cccbe8a314552e3d772411ac37c899" Oct 30 17:13:59.099246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:59.091463 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44" Oct 30 17:13:59.248632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:59.248560 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44"] Oct 30 17:13:59.252886 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:59.252859 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7ebeb6e1-openshift-default-6bd9d9b8d5-8jx44"] Oct 30 17:13:59.289754 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:13:59.289724 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6cb4e0e_750c_4846_ab7c_375ac9fbe956.slice/crio-a0c8d7611c988e7757cef7071182b14d91b59d89243dd67c1062129a4271322a WatchSource:0}: Error finding container a0c8d7611c988e7757cef7071182b14d91b59d89243dd67c1062129a4271322a: Status 404 returned error can't find the container with id a0c8d7611c988e7757cef7071182b14d91b59d89243dd67c1062129a4271322a Oct 30 17:13:59.292922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:13:59.292871 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt"] Oct 30 17:14:00.118870 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:00.118065 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" event={"ID":"c6cb4e0e-750c-4846-ab7c-375ac9fbe956","Type":"ContainerStarted","Data":"fa48fdd1a3858e09c46f254bc77fd2ecb3018d24d2cca5327d59bf009d774445"} Oct 30 17:14:00.118870 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:00.118149 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" event={"ID":"c6cb4e0e-750c-4846-ab7c-375ac9fbe956","Type":"ContainerStarted","Data":"a0c8d7611c988e7757cef7071182b14d91b59d89243dd67c1062129a4271322a"} Oct 30 17:14:00.123336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:00.123141 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:14:00.176248 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:00.176185 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" podStartSLOduration=1.776361027 podStartE2EDuration="2.176165578s" totalImagesPullingTime="399.804551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:00.16986795 +0000 UTC m=+1617.815378134" watchObservedRunningTime="2026-10-30 17:14:00.176165578 +0000 UTC m=+1617.821675759" Oct 30 17:14:00.492175 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:00.492099 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" path="/var/lib/kubelet/pods/836c4dab-6052-49e4-b60e-98bfa0f48b5b/volumes" Oct 30 17:14:00.758986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:00.758514 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:02.599798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:02.599199 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:04.086283 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.086240 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs"] Oct 30 17:14:04.087842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.087794 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="836c4dab-6052-49e4-b60e-98bfa0f48b5b" containerName="istio-proxy" Oct 30 17:14:04.092187 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.092165 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.113337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.113308 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f42baf-serving-cert\"" Oct 30 17:14:04.121753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.121718 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs"] Oct 30 17:14:04.142395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.142362 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-maas-api-tls\") pod \"maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.142685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.142658 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x59rv\" (UniqueName: \"kubernetes.io/projected/3eb31573-e163-4192-9c08-e582a0bad819-kube-api-access-x59rv\") pod \"maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.143023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.142995 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-metrics-certs\") pod \"maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.143253 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.143196 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb31573-e163-4192-9c08-e582a0bad819-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.207075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.207030 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:04.244662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.244630 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-maas-api-tls\") pod \"maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.244855 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.244684 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x59rv\" (UniqueName: \"kubernetes.io/projected/3eb31573-e163-4192-9c08-e582a0bad819-kube-api-access-x59rv\") pod \"maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.244855 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.244814 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-metrics-certs\") pod \"maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.245013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.244909 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb31573-e163-4192-9c08-e582a0bad819-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.245555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.245529 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb31573-e163-4192-9c08-e582a0bad819-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.247167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.247146 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-maas-api-tls\") pod \"maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.247272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.247208 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-metrics-certs\") pod \"maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.318936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.314977 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x59rv\" (UniqueName: \"kubernetes.io/projected/3eb31573-e163-4192-9c08-e582a0bad819-kube-api-access-x59rv\") pod \"maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.411371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.408422 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:04.411371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.409385 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:04.739492 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:14:04.739463 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb31573_e163_4192_9c08_e582a0bad819.slice/crio-f1602ab5c83bd0dd6c70d9338a0d4aa4be8ffc899c895699c1664bde68df3c83 WatchSource:0}: Error finding container f1602ab5c83bd0dd6c70d9338a0d4aa4be8ffc899c895699c1664bde68df3c83: Status 404 returned error can't find the container with id f1602ab5c83bd0dd6c70d9338a0d4aa4be8ffc899c895699c1664bde68df3c83 Oct 30 17:14:04.739911 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:04.739878 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs"] Oct 30 17:14:05.168968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:05.168922 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" event={"ID":"3eb31573-e163-4192-9c08-e582a0bad819","Type":"ContainerStarted","Data":"d45c3802188dfe66bcbae338f1ee380d282d7acd685070ea9294692bff4f1fcd"} Oct 30 17:14:05.169336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:05.169001 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" event={"ID":"3eb31573-e163-4192-9c08-e582a0bad819","Type":"ContainerStarted","Data":"f1602ab5c83bd0dd6c70d9338a0d4aa4be8ffc899c895699c1664bde68df3c83"} Oct 30 17:14:05.219837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:05.219777 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" podStartSLOduration=0.903588978 podStartE2EDuration="1.21975724s" totalImagesPullingTime="316.168262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:05.219130725 +0000 UTC m=+1622.864640905" watchObservedRunningTime="2026-10-30 17:14:05.21975724 +0000 UTC m=+1622.865267443" Oct 30 17:14:05.271368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:05.271331 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:06.012312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.012193 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf"] Oct 30 17:14:06.086671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.078027 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf"] Oct 30 17:14:06.086671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.078168 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.109929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.109264 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-78c9c2-openshift-default-dockercfg-2lskh\"" Oct 30 17:14:06.123937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.122350 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-data\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.123937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.122567 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f58c09-e14a-41e4-a355-c08c7575c648-istiod-ca-cert\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.215961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.169979 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74mv7\" (UniqueName: \"kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-kube-api-access-74mv7\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.215961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.170061 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f58c09-e14a-41e4-a355-c08c7575c648-istio-podinfo\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.215961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.170109 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-certs\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.215961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.178928 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-istio-token\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.215961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.179078 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-envoy\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.215961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.179155 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-credential-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.215961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.179274 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.215961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.180233 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:06.281982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.280945 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-credential-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.281982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.281071 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.281982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.281112 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-data\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.281982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.281319 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f58c09-e14a-41e4-a355-c08c7575c648-istiod-ca-cert\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.281982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.281353 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74mv7\" (UniqueName: \"kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-kube-api-access-74mv7\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.281982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.281392 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f58c09-e14a-41e4-a355-c08c7575c648-istio-podinfo\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.281982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.281428 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-certs\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.281982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.281482 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-istio-token\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.281982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.281577 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-envoy\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.327145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.285989 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f58c09-e14a-41e4-a355-c08c7575c648-istiod-ca-cert\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.327145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.286265 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-credential-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.327145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.286485 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-socket\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.327145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.286706 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-data\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.327145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.286973 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-certs\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.327145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.292422 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f58c09-e14a-41e4-a355-c08c7575c648-istio-podinfo\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.327145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.308452 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:06.327145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.311315 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-envoy\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.335874 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.335824 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74mv7\" (UniqueName: \"kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-kube-api-access-74mv7\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.336103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.335942 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-istio-token\") pod \"e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.422949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.421510 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:06.606190 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.606095 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:06.691925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.689484 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:06.984199 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:14:06.984166 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f58c09_e14a_41e4_a355_c08c7575c648.slice/crio-dc30a213614c6697e52b54aa4df719cca7efb32573af4f32fd207b483ca8093f WatchSource:0}: Error finding container dc30a213614c6697e52b54aa4df719cca7efb32573af4f32fd207b483ca8093f: Status 404 returned error can't find the container with id dc30a213614c6697e52b54aa4df719cca7efb32573af4f32fd207b483ca8093f Oct 30 17:14:06.985783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.985755 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf"] Oct 30 17:14:06.987807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.987762 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:06.987952 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.987853 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:06.987952 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:06.987895 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:07.146210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:07.146175 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:14:07.341146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:07.341104 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" event={"ID":"15f58c09-e14a-41e4-a355-c08c7575c648","Type":"ContainerStarted","Data":"3eecf2d5680e61f434ed7d8651f875ace7b03a6030cb64842a3dbf097a10981f"} Oct 30 17:14:07.341629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:07.341166 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" event={"ID":"15f58c09-e14a-41e4-a355-c08c7575c648","Type":"ContainerStarted","Data":"dc30a213614c6697e52b54aa4df719cca7efb32573af4f32fd207b483ca8093f"} Oct 30 17:14:07.616015 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:07.615317 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:07.913702 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:14:07.913610 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:14:08.534251 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:08.534207 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:08.630925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:08.626884 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:08.724089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:08.724034 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" podStartSLOduration=3.724018638 podStartE2EDuration="3.724018638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:07.407160323 +0000 UTC m=+1625.052670504" watchObservedRunningTime="2026-10-30 17:14:08.724018638 +0000 UTC m=+1626.369528891" Oct 30 17:14:09.358063 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:09.358031 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:09.359238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:09.359212 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:14:10.006732 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:10.006690 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:10.307372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:10.307328 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:10.809935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:10.796332 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:12.055923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:12.055247 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:12.290923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:12.283284 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:12.400068 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:12.395178 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:14:13.246721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.246685 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:13.405437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.405401 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:13.666926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.653989 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7"] Oct 30 17:14:13.681771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.672141 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.710987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.710946 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2d01ec-openshift-default-dockercfg-g2l4d\"" Oct 30 17:14:13.783246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.756016 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:13.826935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.815376 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-socket\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.826935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.815596 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-podinfo\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.826935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.815656 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-credential-socket\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.826935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.815693 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-data\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.826935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.815775 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r629\" (UniqueName: \"kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-kube-api-access-2r629\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.826935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.815819 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-token\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.826935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.815860 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-certs\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.826935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.815927 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d18657c-89fd-4f95-af19-096ef42f4e18-istiod-ca-cert\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.826935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.815977 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-envoy\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.961978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.929310 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-credential-socket\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.961978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.929372 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-data\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.961978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.929457 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r629\" (UniqueName: \"kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-kube-api-access-2r629\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.961978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.929505 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-token\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.961978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.929546 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-certs\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.961978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.929592 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d18657c-89fd-4f95-af19-096ef42f4e18-istiod-ca-cert\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.961978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.929643 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-envoy\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.961978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.929750 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-socket\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.961978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.929983 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-podinfo\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.969923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.969736 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-credential-socket\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:13.970116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.970049 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-data\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:14.017336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.970959 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d18657c-89fd-4f95-af19-096ef42f4e18-istiod-ca-cert\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:14.017336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:13.971192 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-certs\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:14.035252 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:14.025609 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-socket\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:14.076618 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:14.076569 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7"] Oct 30 17:14:14.210853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:14.124145 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-podinfo\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:14.229799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:14.229674 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-envoy\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:14.231978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:14.230811 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r629\" (UniqueName: \"kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-kube-api-access-2r629\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:14.282318 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:14.282254 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:14.299381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:14.297655 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-token\") pod \"e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:14.560931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:14.551348 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:15.695809 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:15.694688 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7"] Oct 30 17:14:15.712122 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:15.712080 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:15.713578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:15.713556 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:15.713727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:15.713708 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:15.995034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:15.991050 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:16.606173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:16.606131 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" event={"ID":"0d18657c-89fd-4f95-af19-096ef42f4e18","Type":"ContainerStarted","Data":"8432fc75d285267ad575baa2b0a8632340a5e625971a1d58567182113bd726ed"} Oct 30 17:14:16.606390 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:16.606198 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" event={"ID":"0d18657c-89fd-4f95-af19-096ef42f4e18","Type":"ContainerStarted","Data":"078fc25286dc8fd91ba468da0e30977e4d9c2225d22854553f1c4a6220667389"} Oct 30 17:14:16.673970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:16.673873 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" podStartSLOduration=3.673859701 podStartE2EDuration="3.673859701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:16.672561113 +0000 UTC m=+1634.318071293" watchObservedRunningTime="2026-10-30 17:14:16.673859701 +0000 UTC m=+1634.319369880" Oct 30 17:14:17.461232 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.461198 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8"] Oct 30 17:14:17.461685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.461453 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" podUID="77de8b26-73dc-4658-aec4-4e582950e6de" containerName="maas-api" containerID="cri-o://860865cebbbdde6bf95846b06bfc60de2b5dff37ee23367b1710fc067ef76af7" gracePeriod=30 Oct 30 17:14:17.552146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.552064 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:17.559190 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.559159 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:17.610641 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.610607 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:17.623566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.623537 4828 generic.go:356] "Generic (PLEG): container finished" podID="77de8b26-73dc-4658-aec4-4e582950e6de" containerID="860865cebbbdde6bf95846b06bfc60de2b5dff37ee23367b1710fc067ef76af7" exitCode=0 Oct 30 17:14:17.623947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.623889 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" event={"ID":"77de8b26-73dc-4658-aec4-4e582950e6de","Type":"ContainerDied","Data":"860865cebbbdde6bf95846b06bfc60de2b5dff37ee23367b1710fc067ef76af7"} Oct 30 17:14:17.624023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.623989 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:17.625320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.625289 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:14:17.788397 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.788370 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:14:17.844995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.844947 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77de8b26-73dc-4658-aec4-4e582950e6de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77de8b26-73dc-4658-aec4-4e582950e6de-openshift-service-ca\") pod \"77de8b26-73dc-4658-aec4-4e582950e6de\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " Oct 30 17:14:17.845239 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.845127 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77de8b26-73dc-4658-aec4-4e582950e6de-kube-api-access-znlw6\" (UniqueName: \"kubernetes.io/projected/77de8b26-73dc-4658-aec4-4e582950e6de-kube-api-access-znlw6\") pod \"77de8b26-73dc-4658-aec4-4e582950e6de\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " Oct 30 17:14:17.845239 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.845190 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-maas-api-tls\") pod \"77de8b26-73dc-4658-aec4-4e582950e6de\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " Oct 30 17:14:17.845387 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.845368 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-metrics-certs\") pod \"77de8b26-73dc-4658-aec4-4e582950e6de\" (UID: \"77de8b26-73dc-4658-aec4-4e582950e6de\") " Oct 30 17:14:17.846092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.846065 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77de8b26-73dc-4658-aec4-4e582950e6de-openshift-service-ca" pod "77de8b26-73dc-4658-aec4-4e582950e6de" (UID: "77de8b26-73dc-4658-aec4-4e582950e6de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:17.848038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.847985 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77de8b26-73dc-4658-aec4-4e582950e6de-kube-api-access-znlw6" pod "77de8b26-73dc-4658-aec4-4e582950e6de" (UID: "77de8b26-73dc-4658-aec4-4e582950e6de"). InnerVolumeSpecName "kube-api-access-znlw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:17.848038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.847986 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-metrics-certs" pod "77de8b26-73dc-4658-aec4-4e582950e6de" (UID: "77de8b26-73dc-4658-aec4-4e582950e6de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:17.849301 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.849274 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-maas-api-tls" pod "77de8b26-73dc-4658-aec4-4e582950e6de" (UID: "77de8b26-73dc-4658-aec4-4e582950e6de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:17.947430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.947396 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:17.947430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.947427 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77de8b26-73dc-4658-aec4-4e582950e6de-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:17.947430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.947437 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znlw6\" (UniqueName: \"kubernetes.io/projected/77de8b26-73dc-4658-aec4-4e582950e6de-kube-api-access-znlw6\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:17.947676 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:17.947446 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77de8b26-73dc-4658-aec4-4e582950e6de-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.230941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:18.228500 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:18.528845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:18.528805 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:18.687871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:18.670661 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" Oct 30 17:14:18.691632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:18.690463 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8" event={"ID":"77de8b26-73dc-4658-aec4-4e582950e6de","Type":"ContainerDied","Data":"074636cdb0243501b42d00fc59ab8640ec90cebc6b9ed83946b42381eba9b7bf"} Oct 30 17:14:18.691632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:18.690559 4828 scope.go:122] "RemoveContainer" containerID="860865cebbbdde6bf95846b06bfc60de2b5dff37ee23367b1710fc067ef76af7" Oct 30 17:14:18.716001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:18.715131 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:18.753890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:18.753854 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8"] Oct 30 17:14:18.766540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:18.766500 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4c2e4-54dc6b564-j9tm8"] Oct 30 17:14:19.108545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:19.108494 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:19.830534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:19.830503 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:20.014947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:20.014892 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:20.493495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:20.493463 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77de8b26-73dc-4658-aec4-4e582950e6de" path="/var/lib/kubelet/pods/77de8b26-73dc-4658-aec4-4e582950e6de/volumes" Oct 30 17:14:20.863621 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:20.863572 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:21.593743 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.593697 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:21.749794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.749764 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb"] Oct 30 17:14:21.750617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.750599 4828 cpu_manager.go:414] "containerMap: removing container" podUID="77de8b26-73dc-4658-aec4-4e582950e6de" containerName="maas-api" Oct 30 17:14:21.750667 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.750619 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77de8b26-73dc-4658-aec4-4e582950e6de" containerName="maas-api" Oct 30 17:14:21.750765 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.750756 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="77de8b26-73dc-4658-aec4-4e582950e6de" containerName="maas-api" Oct 30 17:14:21.753236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.753212 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:21.756027 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.756010 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" Oct 30 17:14:21.780961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.776489 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2d01ec\"" Oct 30 17:14:21.780961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.780230 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb"] Oct 30 17:14:21.784777 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.784672 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2d01ec-dockercfg-nqvkh\"" Oct 30 17:14:21.824442 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.824402 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3138d-b28b-4d4f-9173-48d05241873f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb\" (UID: \"55c3138d-b28b-4d4f-9173-48d05241873f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" Oct 30 17:14:21.824736 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.824708 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlldr\" (UniqueName: \"kubernetes.io/projected/55c3138d-b28b-4d4f-9173-48d05241873f-kube-api-access-dlldr\") pod \"payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb\" (UID: \"55c3138d-b28b-4d4f-9173-48d05241873f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" Oct 30 17:14:21.947974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.931832 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3138d-b28b-4d4f-9173-48d05241873f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb\" (UID: \"55c3138d-b28b-4d4f-9173-48d05241873f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" Oct 30 17:14:21.947974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.932090 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlldr\" (UniqueName: \"kubernetes.io/projected/55c3138d-b28b-4d4f-9173-48d05241873f-kube-api-access-dlldr\") pod \"payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb\" (UID: \"55c3138d-b28b-4d4f-9173-48d05241873f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" Oct 30 17:14:21.947974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:21.943238 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3138d-b28b-4d4f-9173-48d05241873f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb\" (UID: \"55c3138d-b28b-4d4f-9173-48d05241873f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" Oct 30 17:14:22.153357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.153279 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlldr\" (UniqueName: \"kubernetes.io/projected/55c3138d-b28b-4d4f-9173-48d05241873f-kube-api-access-dlldr\") pod \"payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb\" (UID: \"55c3138d-b28b-4d4f-9173-48d05241873f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" Oct 30 17:14:22.267290 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.267245 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh"] Oct 30 17:14:22.281987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.279301 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh"] Oct 30 17:14:22.281987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.279402 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.316129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.316099 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-39fecf-openshift-default-dockercfg-8ldpr\"" Oct 30 17:14:22.371532 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.371490 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" Oct 30 17:14:22.383378 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.383334 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-certs\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.383529 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.383508 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-token\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.383678 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.383649 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-data\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.383768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.383750 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx674\" (UniqueName: \"kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-kube-api-access-hx674\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.383837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.383820 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-credential-socket\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.383945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.383881 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-podinfo\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.384009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.383947 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-socket\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.384009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.383978 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-envoy\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.384127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.384101 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istiod-ca-cert\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.486860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.486004 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-socket\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.486860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.486043 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-envoy\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.486860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.486089 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istiod-ca-cert\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.486860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.486227 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-certs\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.486860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.486280 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-token\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.486860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.486328 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-data\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.486860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.486372 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx674\" (UniqueName: \"kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-kube-api-access-hx674\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.486860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.486412 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-credential-socket\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.486860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.486430 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-podinfo\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.486860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.486590 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-socket\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.488664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.487858 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-data\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.488664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.488391 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-credential-socket\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.489450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.488929 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-certs\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.489450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.489399 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istiod-ca-cert\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.489664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.489638 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-envoy\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.490951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.490880 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-podinfo\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.514142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.514100 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx674\" (UniqueName: \"kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-kube-api-access-hx674\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.514952 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.514883 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-token\") pod \"e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.602249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.602165 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:22.709166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.705777 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb"] Oct 30 17:14:22.710815 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:14:22.710773 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c3138d_b28b_4d4f_9173_48d05241873f.slice/crio-0c62f09cd76a3183ff340c40e4b193692fb15d2a74326d47cc5d69329891aed4 WatchSource:0}: Error finding container 0c62f09cd76a3183ff340c40e4b193692fb15d2a74326d47cc5d69329891aed4: Status 404 returned error can't find the container with id 0c62f09cd76a3183ff340c40e4b193692fb15d2a74326d47cc5d69329891aed4 Oct 30 17:14:22.924822 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.924795 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh"] Oct 30 17:14:22.926675 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:14:22.926647 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe67cd22_adb6_4549_8a92_7a86b91cd1ac.slice/crio-ce028d65855e38944ede6ab17614e824d55ccdb3f4c2b5edc716b2560087ed2f WatchSource:0}: Error finding container ce028d65855e38944ede6ab17614e824d55ccdb3f4c2b5edc716b2560087ed2f: Status 404 returned error can't find the container with id ce028d65855e38944ede6ab17614e824d55ccdb3f4c2b5edc716b2560087ed2f Oct 30 17:14:22.929480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.929449 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:22.929548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.929513 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:22.929548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.929541 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:22.960351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:22.957788 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:23.730479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:23.730440 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" event={"ID":"fe67cd22-adb6-4549-8a92-7a86b91cd1ac","Type":"ContainerStarted","Data":"84311205c96841b56b7559db031452672b43058c252846bbdea84b63fd1e2d58"} Oct 30 17:14:23.730712 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:23.730504 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" event={"ID":"fe67cd22-adb6-4549-8a92-7a86b91cd1ac","Type":"ContainerStarted","Data":"ce028d65855e38944ede6ab17614e824d55ccdb3f4c2b5edc716b2560087ed2f"} Oct 30 17:14:23.733077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:23.732955 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" event={"ID":"55c3138d-b28b-4d4f-9173-48d05241873f","Type":"ContainerStarted","Data":"609a7fe64677fa57d3787e20b3e6f99785bbae7b05c3cf4f772aaf6550957287"} Oct 30 17:14:23.733077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:23.733022 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" event={"ID":"55c3138d-b28b-4d4f-9173-48d05241873f","Type":"ContainerStarted","Data":"0c62f09cd76a3183ff340c40e4b193692fb15d2a74326d47cc5d69329891aed4"} Oct 30 17:14:23.733077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:23.733064 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" Oct 30 17:14:23.794923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:23.783783 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" podStartSLOduration=1.78376364 podStartE2EDuration="1.78376364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:23.780628303 +0000 UTC m=+1641.426138484" watchObservedRunningTime="2026-10-30 17:14:23.78376364 +0000 UTC m=+1641.429273822" Oct 30 17:14:24.603178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:24.603079 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:24.604546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:24.604514 4828 patch_prober.go:29] interesting pod/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 17:14:24.604651 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:24.604571 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" podUID="fe67cd22-adb6-4549-8a92-7a86b91cd1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 17:14:24.831924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:24.831649 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:25.603662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:25.603157 4828 patch_prober.go:29] interesting pod/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 17:14:25.603662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:25.603214 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" podUID="fe67cd22-adb6-4549-8a92-7a86b91cd1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 17:14:26.608348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:26.608307 4828 patch_prober.go:29] interesting pod/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 17:14:26.608954 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:26.608920 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" podUID="fe67cd22-adb6-4549-8a92-7a86b91cd1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 17:14:26.943971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:26.940451 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:27.119560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:27.118645 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:27.351948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:27.350378 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" podStartSLOduration=6.350362888 podStartE2EDuration="6.350362888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:23.834818063 +0000 UTC m=+1641.480328267" watchObservedRunningTime="2026-10-30 17:14:27.350362888 +0000 UTC m=+1644.995873101" Oct 30 17:14:27.351948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:27.351076 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8"] Oct 30 17:14:27.351948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:27.351290 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" containerName="istio-proxy" containerID="cri-o://b81fc983f0dbd3c9512e35fd29f893e7ca544198c40c3259973c373bbbb294aa" gracePeriod=30 Oct 30 17:14:27.419405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:27.419369 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:27.624134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:27.624043 4828 patch_prober.go:29] interesting pod/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 17:14:27.624694 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:27.624659 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" podUID="fe67cd22-adb6-4549-8a92-7a86b91cd1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 17:14:28.284579 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.284540 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:28.604714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.604313 4828 patch_prober.go:29] interesting pod/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 17:14:28.604714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.604378 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" podUID="fe67cd22-adb6-4549-8a92-7a86b91cd1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 17:14:28.757632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.757588 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt"] Oct 30 17:14:28.763225 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.763190 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.785923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.782125 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7c52f8-openshift-default-dockercfg-lmxqw\"" Oct 30 17:14:28.825985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.823748 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.825985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.823893 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.825985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.824026 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.825985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.824053 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.825985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.824095 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.825985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.824260 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.825985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.824371 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgckb\" (UniqueName: \"kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-kube-api-access-kgckb\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.825985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.824439 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.825985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.824479 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.831888 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.831853 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt"] Oct 30 17:14:28.917580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.917481 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:28.928005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.927825 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.928005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.927988 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.928312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.928024 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.928312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.928067 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.928312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.928123 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.928312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.928216 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgckb\" (UniqueName: \"kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-kube-api-access-kgckb\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.928312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.928250 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.928312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.928287 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.928717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.928428 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.929929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.929009 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.942026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.941976 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.942195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.942117 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.942195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.942137 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.942400 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.942382 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.944377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.944348 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:28.946359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:28.946334 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:29.050939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.050381 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:29.081927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.076987 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgckb\" (UniqueName: \"kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-kube-api-access-kgckb\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:29.094505 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.094466 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:29.137929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.125224 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:29.535559 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.535272 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:29.602661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.602629 4828 patch_prober.go:29] interesting pod/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 17:14:29.602833 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.602683 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" podUID="fe67cd22-adb6-4549-8a92-7a86b91cd1ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 17:14:29.689027 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:14:29.688797 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7d85b1_1928_43bb_9f10_6a76f1d88a2a.slice/crio-0876774a7b33350c0f2beadd7464f5f6495b91674d941db68e0c1f5ec6e3b016 WatchSource:0}: Error finding container 0876774a7b33350c0f2beadd7464f5f6495b91674d941db68e0c1f5ec6e3b016: Status 404 returned error can't find the container with id 0876774a7b33350c0f2beadd7464f5f6495b91674d941db68e0c1f5ec6e3b016 Oct 30 17:14:29.690482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.690441 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt"] Oct 30 17:14:29.691509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.691476 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:29.691588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.691551 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:29.691588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.691581 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:29.729343 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.729305 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:29.839429 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.839191 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" event={"ID":"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a","Type":"ContainerStarted","Data":"9ff64fd8dd9db293aa8e03fbf2fc3ddc9b3204f7350a532bf9f1bd50a20c04d3"} Oct 30 17:14:29.839429 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.839276 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" event={"ID":"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a","Type":"ContainerStarted","Data":"0876774a7b33350c0f2beadd7464f5f6495b91674d941db68e0c1f5ec6e3b016"} Oct 30 17:14:29.893143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:29.893079 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" podStartSLOduration=1.8930638069999999 podStartE2EDuration="1.893063807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:29.887464228 +0000 UTC m=+1647.532974409" watchObservedRunningTime="2026-10-30 17:14:29.893063807 +0000 UTC m=+1647.538573986" Oct 30 17:14:30.140979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:30.139041 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:30.144533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:30.144501 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 17:14:30.144712 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:30.144592 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 17:14:30.608645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:30.608617 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:30.609158 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:30.609135 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:30.610103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:30.610084 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:14:31.160928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:31.159936 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:31.190873 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:31.190777 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:31.860471 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:31.860438 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:31.861398 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:31.861375 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:14:32.688458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.688423 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:14:32.694279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.694240 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:32.717693 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.717656 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-istio-token\" (UniqueName: \"kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-istio-token\") pod \"068993d6-1de9-4224-b028-39ce4fb563ea\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " Oct 30 17:14:32.717862 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.717704 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-kube-api-access-f4jc6\" (UniqueName: \"kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-kube-api-access-f4jc6\") pod \"068993d6-1de9-4224-b028-39ce4fb563ea\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " Oct 30 17:14:32.717862 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.717789 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-envoy\") pod \"068993d6-1de9-4224-b028-39ce4fb563ea\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " Oct 30 17:14:32.717862 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.717850 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-socket\") pod \"068993d6-1de9-4224-b028-39ce4fb563ea\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " Oct 30 17:14:32.718041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.717992 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-credential-socket\") pod \"068993d6-1de9-4224-b028-39ce4fb563ea\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " Oct 30 17:14:32.718211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.718186 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-certs\") pod \"068993d6-1de9-4224-b028-39ce4fb563ea\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " Oct 30 17:14:32.718320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.718282 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-socket" pod "068993d6-1de9-4224-b028-39ce4fb563ea" (UID: "068993d6-1de9-4224-b028-39ce4fb563ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:32.718375 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.718316 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-data\") pod \"068993d6-1de9-4224-b028-39ce4fb563ea\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " Oct 30 17:14:32.718375 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.718357 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/068993d6-1de9-4224-b028-39ce4fb563ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068993d6-1de9-4224-b028-39ce4fb563ea-istiod-ca-cert\") pod \"068993d6-1de9-4224-b028-39ce4fb563ea\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " Oct 30 17:14:32.718484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.718403 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/068993d6-1de9-4224-b028-39ce4fb563ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068993d6-1de9-4224-b028-39ce4fb563ea-istio-podinfo\") pod \"068993d6-1de9-4224-b028-39ce4fb563ea\" (UID: \"068993d6-1de9-4224-b028-39ce4fb563ea\") " Oct 30 17:14:32.718484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.718406 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-credential-socket" pod "068993d6-1de9-4224-b028-39ce4fb563ea" (UID: "068993d6-1de9-4224-b028-39ce4fb563ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:32.718594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.718516 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-certs" pod "068993d6-1de9-4224-b028-39ce4fb563ea" (UID: "068993d6-1de9-4224-b028-39ce4fb563ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:32.718678 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.718615 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-data" pod "068993d6-1de9-4224-b028-39ce4fb563ea" (UID: "068993d6-1de9-4224-b028-39ce4fb563ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:32.718861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.718839 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068993d6-1de9-4224-b028-39ce4fb563ea-istiod-ca-cert" pod "068993d6-1de9-4224-b028-39ce4fb563ea" (UID: "068993d6-1de9-4224-b028-39ce4fb563ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:32.719193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.719152 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.719193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.719182 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.719372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.719195 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.719372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.719209 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.719372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.719281 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/068993d6-1de9-4224-b028-39ce4fb563ea-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.720484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.720441 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-envoy" pod "068993d6-1de9-4224-b028-39ce4fb563ea" (UID: "068993d6-1de9-4224-b028-39ce4fb563ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:32.721010 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.720984 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/068993d6-1de9-4224-b028-39ce4fb563ea-istio-podinfo" pod "068993d6-1de9-4224-b028-39ce4fb563ea" (UID: "068993d6-1de9-4224-b028-39ce4fb563ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:14:32.721594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.721569 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-istio-token" pod "068993d6-1de9-4224-b028-39ce4fb563ea" (UID: "068993d6-1de9-4224-b028-39ce4fb563ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:32.722475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.722115 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-kube-api-access-f4jc6" pod "068993d6-1de9-4224-b028-39ce4fb563ea" (UID: "068993d6-1de9-4224-b028-39ce4fb563ea"). InnerVolumeSpecName "kube-api-access-f4jc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:32.820696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.820664 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/068993d6-1de9-4224-b028-39ce4fb563ea-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.820696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.820695 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.820957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.820707 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4jc6\" (UniqueName: \"kubernetes.io/projected/068993d6-1de9-4224-b028-39ce4fb563ea-kube-api-access-f4jc6\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.820957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.820721 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/068993d6-1de9-4224-b028-39ce4fb563ea-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:32.871827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.871794 4828 generic.go:356] "Generic (PLEG): container finished" podID="068993d6-1de9-4224-b028-39ce4fb563ea" containerID="b81fc983f0dbd3c9512e35fd29f893e7ca544198c40c3259973c373bbbb294aa" exitCode=0 Oct 30 17:14:32.872011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.871843 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" Oct 30 17:14:32.872011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.871882 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" event={"ID":"068993d6-1de9-4224-b028-39ce4fb563ea","Type":"ContainerDied","Data":"b81fc983f0dbd3c9512e35fd29f893e7ca544198c40c3259973c373bbbb294aa"} Oct 30 17:14:32.872011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.871951 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8" event={"ID":"068993d6-1de9-4224-b028-39ce4fb563ea","Type":"ContainerDied","Data":"c51dff39fb31b1165a9d700c2517277838f28f8e6f1b6976b907a6db0aec3c00"} Oct 30 17:14:32.872011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.871979 4828 scope.go:122] "RemoveContainer" containerID="b81fc983f0dbd3c9512e35fd29f893e7ca544198c40c3259973c373bbbb294aa" Oct 30 17:14:32.885130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.885109 4828 scope.go:122] "RemoveContainer" containerID="aef9d048db08610705874f6def9ab3b811be2246deb4ac2d7995304da7b878a9" Oct 30 17:14:32.895516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.895496 4828 scope.go:122] "RemoveContainer" containerID="b81fc983f0dbd3c9512e35fd29f893e7ca544198c40c3259973c373bbbb294aa" Oct 30 17:14:32.895774 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:14:32.895754 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81fc983f0dbd3c9512e35fd29f893e7ca544198c40c3259973c373bbbb294aa\": container with ID starting with b81fc983f0dbd3c9512e35fd29f893e7ca544198c40c3259973c373bbbb294aa not found: ID does not exist" containerID="b81fc983f0dbd3c9512e35fd29f893e7ca544198c40c3259973c373bbbb294aa" Oct 30 17:14:32.895842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.895782 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81fc983f0dbd3c9512e35fd29f893e7ca544198c40c3259973c373bbbb294aa"} err="failed to get container status \"b81fc983f0dbd3c9512e35fd29f893e7ca544198c40c3259973c373bbbb294aa\": rpc error: code = NotFound desc = could not find container \"b81fc983f0dbd3c9512e35fd29f893e7ca544198c40c3259973c373bbbb294aa\": container with ID starting with b81fc983f0dbd3c9512e35fd29f893e7ca544198c40c3259973c373bbbb294aa not found: ID does not exist" Oct 30 17:14:32.895842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.895800 4828 scope.go:122] "RemoveContainer" containerID="aef9d048db08610705874f6def9ab3b811be2246deb4ac2d7995304da7b878a9" Oct 30 17:14:32.896109 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:14:32.896089 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef9d048db08610705874f6def9ab3b811be2246deb4ac2d7995304da7b878a9\": container with ID starting with aef9d048db08610705874f6def9ab3b811be2246deb4ac2d7995304da7b878a9 not found: ID does not exist" containerID="aef9d048db08610705874f6def9ab3b811be2246deb4ac2d7995304da7b878a9" Oct 30 17:14:32.896177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.896118 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef9d048db08610705874f6def9ab3b811be2246deb4ac2d7995304da7b878a9"} err="failed to get container status \"aef9d048db08610705874f6def9ab3b811be2246deb4ac2d7995304da7b878a9\": rpc error: code = NotFound desc = could not find container \"aef9d048db08610705874f6def9ab3b811be2246deb4ac2d7995304da7b878a9\": container with ID starting with aef9d048db08610705874f6def9ab3b811be2246deb4ac2d7995304da7b878a9 not found: ID does not exist" Oct 30 17:14:32.941938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.941891 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8"] Oct 30 17:14:32.947818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:32.947790 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c4c2e4-openshift-default-5f77dcb8f7-vc5x8"] Oct 30 17:14:33.079624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:33.079579 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:33.540929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:33.539112 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:34.471687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:34.471647 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:34.493825 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:34.493789 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" path="/var/lib/kubelet/pods/068993d6-1de9-4224-b028-39ce4fb563ea/volumes" Oct 30 17:14:34.666923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:34.666836 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:34.804974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:34.802168 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" Oct 30 17:14:35.538821 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:35.538785 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:37.223606 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:37.223573 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:37.897316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:37.865892 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:37.974892 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:14:37.974857 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:14:38.067779 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:38.067741 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:38.284923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:38.267964 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:38.627194 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:38.627111 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:40.484479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:40.484441 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:42.224110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:42.224027 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:44.093955 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:44.093895 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:45.673267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:45.673228 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:46.097166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.097135 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:46.295562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.295521 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:46.694159 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.694098 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8"] Oct 30 17:14:46.696806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.696774 4828 cpu_manager.go:414] "containerMap: removing container" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" containerName="istio-proxy" Oct 30 17:14:46.696806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.696806 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" containerName="istio-proxy" Oct 30 17:14:46.697029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.696846 4828 cpu_manager.go:414] "containerMap: removing container" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" containerName="istio-proxy" Oct 30 17:14:46.697029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.696855 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" containerName="istio-proxy" Oct 30 17:14:46.697104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.697091 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" containerName="istio-proxy" Oct 30 17:14:46.697142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.697111 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="068993d6-1de9-4224-b028-39ce4fb563ea" containerName="istio-proxy" Oct 30 17:14:46.703641 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.703617 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:46.717671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.717633 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8"] Oct 30 17:14:46.720265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.720222 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c82666-openshift-default-dockercfg-z7xj6\"" Oct 30 17:14:46.772408 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.772253 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:46.840957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.826069 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istiod-ca-cert\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:46.869960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.867954 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-socket\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:46.870309 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.870284 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-data\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:46.870463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.870446 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-credential-socket\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:46.870641 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.870627 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-certs\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:46.870884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.870868 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2z7v\" (UniqueName: \"kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-kube-api-access-s2z7v\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:46.884922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.871117 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-envoy\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:46.885310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.885280 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-podinfo\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:46.885447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.885432 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-token\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.015971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.988000 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istiod-ca-cert\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.016306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.016270 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-socket\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.016458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.016442 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-data\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.016584 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.016571 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-credential-socket\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.016762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.016748 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-certs\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.017037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.017021 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2z7v\" (UniqueName: \"kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-kube-api-access-s2z7v\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.017230 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.017212 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-envoy\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.017405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.017390 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-podinfo\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.017512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.017499 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-token\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.018134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:46.998808 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istiod-ca-cert\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.041987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.036445 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-certs\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.042227 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.036761 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-socket\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.042366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.037063 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-data\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.042464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.037323 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-credential-socket\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.129453 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.113677 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-podinfo\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.150374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.135203 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-envoy\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.150374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.138941 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:47.389976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.375549 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:47.403013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.400160 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2z7v\" (UniqueName: \"kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-kube-api-access-s2z7v\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.421922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.417663 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9"] Oct 30 17:14:47.421922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.418365 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" containerName="istio-proxy" containerID="cri-o://0aa541caae3a8feb1f8f5bbced6684411c2ea647a4d84eb2ff5cd368fc61cc06" gracePeriod=30 Oct 30 17:14:47.437189 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.432754 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-token\") pod \"e2e-inf-b-c82666-openshift-default-767d987659-7j4f8\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:47.595934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.594095 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:47.641862 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:47.641525 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:48.145149 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:14:48.145113 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330fe0a1_61b1_42d0_a2d9_ca7e9b3ce1d0.slice/crio-e8181dbd09c8b7608dbc4fa05527db4c0009935ed745b55352f3bfee26f1ae88 WatchSource:0}: Error finding container e8181dbd09c8b7608dbc4fa05527db4c0009935ed745b55352f3bfee26f1ae88: Status 404 returned error can't find the container with id e8181dbd09c8b7608dbc4fa05527db4c0009935ed745b55352f3bfee26f1ae88 Oct 30 17:14:48.146459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:48.146436 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8"] Oct 30 17:14:48.148517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:48.148480 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:48.148596 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:48.148562 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:48.148646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:48.148630 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:14:48.254625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:48.254586 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22"] Oct 30 17:14:48.344631 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:48.344596 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:48.433816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:48.433713 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" event={"ID":"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0","Type":"ContainerStarted","Data":"daa4cf24bfd00b221e1daa64ff4584e2ff59118ee1b77d95d2572060caf4bac2"} Oct 30 17:14:48.434091 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:48.434067 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" event={"ID":"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0","Type":"ContainerStarted","Data":"e8181dbd09c8b7608dbc4fa05527db4c0009935ed745b55352f3bfee26f1ae88"} Oct 30 17:14:48.564049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:48.535353 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" podStartSLOduration=2.535332878 podStartE2EDuration="2.535332878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:48.488413966 +0000 UTC m=+1666.133924158" watchObservedRunningTime="2026-10-30 17:14:48.535332878 +0000 UTC m=+1666.180843059" Oct 30 17:14:48.691217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:48.691123 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:49.184925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:49.175548 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:49.649609 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:49.649529 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:50.449775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:50.449749 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:50.450986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:50.450963 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:14:51.076545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:51.076487 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:51.239923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:51.237505 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:52.581529 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.581497 4828 generic.go:356] "Generic (PLEG): container finished" podID="e2bbc703-7b2d-4c94-8730-63135103703e" containerID="0aa541caae3a8feb1f8f5bbced6684411c2ea647a4d84eb2ff5cd368fc61cc06" exitCode=0 Oct 30 17:14:52.582054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.581560 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" event={"ID":"e2bbc703-7b2d-4c94-8730-63135103703e","Type":"ContainerDied","Data":"0aa541caae3a8feb1f8f5bbced6684411c2ea647a4d84eb2ff5cd368fc61cc06"} Oct 30 17:14:52.582054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.581740 4828 scope.go:122] "RemoveContainer" containerID="01084f1d8eea4c1d0c123590df00d671ac525ab23d5407a94c53f030547f3d2a" Oct 30 17:14:52.772105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.772050 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:14:52.778826 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.778763 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:52.835472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.835431 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-data\") pod \"e2bbc703-7b2d-4c94-8730-63135103703e\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " Oct 30 17:14:52.835672 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.835629 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-envoy\") pod \"e2bbc703-7b2d-4c94-8730-63135103703e\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " Oct 30 17:14:52.835731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.835670 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2bbc703-7b2d-4c94-8730-63135103703e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2bbc703-7b2d-4c94-8730-63135103703e-istiod-ca-cert\") pod \"e2bbc703-7b2d-4c94-8730-63135103703e\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " Oct 30 17:14:52.835731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.835708 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-certs\") pod \"e2bbc703-7b2d-4c94-8730-63135103703e\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " Oct 30 17:14:52.835844 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.835808 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-kube-api-access-9hx8k\" (UniqueName: \"kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-kube-api-access-9hx8k\") pod \"e2bbc703-7b2d-4c94-8730-63135103703e\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " Oct 30 17:14:52.835844 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.835811 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-data" pod "e2bbc703-7b2d-4c94-8730-63135103703e" (UID: "e2bbc703-7b2d-4c94-8730-63135103703e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:52.836006 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.835932 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-credential-socket\") pod \"e2bbc703-7b2d-4c94-8730-63135103703e\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " Oct 30 17:14:52.836006 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.835965 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-socket\") pod \"e2bbc703-7b2d-4c94-8730-63135103703e\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " Oct 30 17:14:52.836114 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.836019 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2bbc703-7b2d-4c94-8730-63135103703e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2bbc703-7b2d-4c94-8730-63135103703e-istio-podinfo\") pod \"e2bbc703-7b2d-4c94-8730-63135103703e\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " Oct 30 17:14:52.836114 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.836109 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-istio-token\" (UniqueName: \"kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-istio-token\") pod \"e2bbc703-7b2d-4c94-8730-63135103703e\" (UID: \"e2bbc703-7b2d-4c94-8730-63135103703e\") " Oct 30 17:14:52.836226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.836159 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2bbc703-7b2d-4c94-8730-63135103703e-istiod-ca-cert" pod "e2bbc703-7b2d-4c94-8730-63135103703e" (UID: "e2bbc703-7b2d-4c94-8730-63135103703e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:52.836616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.836587 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-credential-socket" pod "e2bbc703-7b2d-4c94-8730-63135103703e" (UID: "e2bbc703-7b2d-4c94-8730-63135103703e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:52.837300 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.836953 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:52.837300 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.836980 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:52.837300 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.836994 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2bbc703-7b2d-4c94-8730-63135103703e-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:52.837300 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.837198 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-certs" pod "e2bbc703-7b2d-4c94-8730-63135103703e" (UID: "e2bbc703-7b2d-4c94-8730-63135103703e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:52.838273 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.837982 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-socket" pod "e2bbc703-7b2d-4c94-8730-63135103703e" (UID: "e2bbc703-7b2d-4c94-8730-63135103703e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:52.845558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.844143 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2bbc703-7b2d-4c94-8730-63135103703e-istio-podinfo" pod "e2bbc703-7b2d-4c94-8730-63135103703e" (UID: "e2bbc703-7b2d-4c94-8730-63135103703e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:14:52.845558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.844169 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-envoy" pod "e2bbc703-7b2d-4c94-8730-63135103703e" (UID: "e2bbc703-7b2d-4c94-8730-63135103703e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:52.845558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.844207 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-istio-token" pod "e2bbc703-7b2d-4c94-8730-63135103703e" (UID: "e2bbc703-7b2d-4c94-8730-63135103703e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:52.845558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.844848 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-kube-api-access-9hx8k" pod "e2bbc703-7b2d-4c94-8730-63135103703e" (UID: "e2bbc703-7b2d-4c94-8730-63135103703e"). InnerVolumeSpecName "kube-api-access-9hx8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:52.938079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.938045 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:52.938079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.938077 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:52.938289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.938092 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hx8k\" (UniqueName: \"kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-kube-api-access-9hx8k\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:52.938289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.938108 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2bbc703-7b2d-4c94-8730-63135103703e-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:52.938289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.938121 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2bbc703-7b2d-4c94-8730-63135103703e-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:52.938289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.938132 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2bbc703-7b2d-4c94-8730-63135103703e-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:14:52.942150 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:52.942114 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:53.109930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.103948 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6"] Oct 30 17:14:53.109930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.105219 4828 cpu_manager.go:414] "containerMap: removing container" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" containerName="istio-proxy" Oct 30 17:14:53.109930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.105242 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" containerName="istio-proxy" Oct 30 17:14:53.109930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.105333 4828 cpu_manager.go:414] "containerMap: removing container" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" containerName="istio-proxy" Oct 30 17:14:53.109930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.105340 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" containerName="istio-proxy" Oct 30 17:14:53.109930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.105502 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" containerName="istio-proxy" Oct 30 17:14:53.134923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.127477 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.152925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.151254 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-78c9c2-serving-cert\"" Oct 30 17:14:53.186927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.177026 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6"] Oct 30 17:14:53.275740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.274739 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16084cf-b957-48f1-8c87-cf83e016da52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.275740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.274829 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wbh2\" (UniqueName: \"kubernetes.io/projected/e16084cf-b957-48f1-8c87-cf83e016da52-kube-api-access-5wbh2\") pod \"maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.275740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.274955 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.275740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.275071 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.393183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.393089 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.393633 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.393607 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16084cf-b957-48f1-8c87-cf83e016da52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.393849 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.393832 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wbh2\" (UniqueName: \"kubernetes.io/projected/e16084cf-b957-48f1-8c87-cf83e016da52-kube-api-access-5wbh2\") pod \"maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.395444 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.395419 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.397649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.397023 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16084cf-b957-48f1-8c87-cf83e016da52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.407004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.402345 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.407860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.407799 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.422334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.422226 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:53.422975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.422950 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wbh2\" (UniqueName: \"kubernetes.io/projected/e16084cf-b957-48f1-8c87-cf83e016da52-kube-api-access-5wbh2\") pod \"maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.556301 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.556257 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:53.630838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.630792 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:53.684720 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.680304 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" event={"ID":"e2bbc703-7b2d-4c94-8730-63135103703e","Type":"ContainerDied","Data":"3eb24d9e6aa0f8ed7da12dd482cb7edf467cb42835bc31d50414c12f4bc7f3ba"} Oct 30 17:14:53.684720 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.680413 4828 scope.go:122] "RemoveContainer" containerID="0aa541caae3a8feb1f8f5bbced6684411c2ea647a4d84eb2ff5cd368fc61cc06" Oct 30 17:14:53.684720 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.680578 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9" Oct 30 17:14:53.754953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.754400 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9"] Oct 30 17:14:53.769805 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.766710 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-432680-openshift-default-68fdbd8df9-l2hw9"] Oct 30 17:14:53.985353 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:53.985320 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6"] Oct 30 17:14:54.086455 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:54.086420 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:54.504052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:54.503969 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" path="/var/lib/kubelet/pods/e2bbc703-7b2d-4c94-8730-63135103703e/volumes" Oct 30 17:14:54.563207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:54.563176 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:54.693914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:54.693866 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" event={"ID":"e16084cf-b957-48f1-8c87-cf83e016da52","Type":"ContainerStarted","Data":"aac5f7d87c02c3ff238c36a4d222b8150974191803b4486db4d81be90a0324eb"} Oct 30 17:14:54.694360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:54.693946 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" event={"ID":"e16084cf-b957-48f1-8c87-cf83e016da52","Type":"ContainerStarted","Data":"cd6b77c30d6621be258111002336299e6d2184e3060867a2d2208ee91cc8a369"} Oct 30 17:14:54.694360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:54.693971 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:14:54.738351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:54.738254 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" podStartSLOduration=1.252263428 podStartE2EDuration="1.738239425s" totalImagesPullingTime="485.975997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:54.737487739 +0000 UTC m=+1672.382997931" watchObservedRunningTime="2026-10-30 17:14:54.738239425 +0000 UTC m=+1672.383749606" Oct 30 17:14:55.235924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:55.229842 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:55.521920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:55.517226 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:55.879988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:55.878740 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:56.850727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:56.847339 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:58.603603 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:58.603563 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:58.886009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:58.884599 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.145:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:14:58.922368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:58.922325 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:59.177563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:59.177470 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:14:59.394923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:59.386241 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:14:59.712078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:59.712031 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:14:59.793935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:59.793867 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:14:59.794329 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:59.793887 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:14:59.945923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:59.945039 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.145:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:14:59.945923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:14:59.945728 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:15:00.188935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:00.185328 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:15:00.212932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:00.211494 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:15:00.389525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:00.388026 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:00.389525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:00.388612 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:15:00.810043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:00.810000 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:00.864936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:00.859283 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:00.898072 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:00.898039 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:15:00.912790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:00.912757 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:15:00.933586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:00.933552 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc"] Oct 30 17:15:00.956640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:00.956182 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2bbc703-7b2d-4c94-8730-63135103703e" containerName="istio-proxy" Oct 30 17:15:00.976933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:00.975138 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" Oct 30 17:15:01.010247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.009754 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:15:01.058797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.055095 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:15:01.114443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.111359 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99gr5\" (UniqueName: \"kubernetes.io/projected/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-kube-api-access-99gr5\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc\" (UID: \"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" Oct 30 17:15:01.114443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.111508 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc\" (UID: \"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" Oct 30 17:15:01.260928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.241787 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99gr5\" (UniqueName: \"kubernetes.io/projected/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-kube-api-access-99gr5\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc\" (UID: \"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" Oct 30 17:15:01.298926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.297158 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc\" (UID: \"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" Oct 30 17:15:01.298926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.277171 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:01.301929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.299223 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc\" (UID: \"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" Oct 30 17:15:01.301929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.277724 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:01.396651 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.320354 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:15:01.396651 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.345913 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:01.409030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.399531 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:01.646159 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.646094 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:01.668250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.668158 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc"] Oct 30 17:15:01.796924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.763168 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99gr5\" (UniqueName: \"kubernetes.io/projected/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-kube-api-access-99gr5\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc\" (UID: \"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" Oct 30 17:15:01.836174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.836128 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": context deadline exceeded" Oct 30 17:15:01.867931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.866330 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:01.939284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.938315 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph"] Oct 30 17:15:01.978612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:01.956511 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:15:02.077950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:02.068225 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" Oct 30 17:15:02.244886 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:02.244795 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.145:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:02.306092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:02.299013 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:15:02.404799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:02.402366 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:15:02.417178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:02.409273 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:02.437670 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:02.437634 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:03.029928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:03.028455 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:15:03.167915 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:15:03.167868 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec6ba67_fcac_4b8f_bfc2_ad7dc83786d1.slice/crio-99e55baae9ae6694d43317f0572e7ebca0b636b0a34d8e21a5f4b34622701d3b WatchSource:0}: Error finding container 99e55baae9ae6694d43317f0572e7ebca0b636b0a34d8e21a5f4b34622701d3b: Status 404 returned error can't find the container with id 99e55baae9ae6694d43317f0572e7ebca0b636b0a34d8e21a5f4b34622701d3b Oct 30 17:15:03.172924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:03.169627 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc"] Oct 30 17:15:03.299000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:03.298108 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerName="main" containerID="cri-o://1314121c0bebb10cf5e89b39b43f1226d204fad1e8bc2f3f3d1517f883568f2d" gracePeriod=45 Oct 30 17:15:03.959314 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:03.958393 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:15:04.050211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.050172 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:04.050787 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.050386 4828 generic.go:356] "Generic (PLEG): container finished" podID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerID="1314121c0bebb10cf5e89b39b43f1226d204fad1e8bc2f3f3d1517f883568f2d" exitCode=0 Oct 30 17:15:04.051038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.050423 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" event={"ID":"9875cf06-8438-4b8e-96b2-592d1ea387bf","Type":"ContainerDied","Data":"1314121c0bebb10cf5e89b39b43f1226d204fad1e8bc2f3f3d1517f883568f2d"} Oct 30 17:15:04.051195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.051174 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" event={"ID":"9875cf06-8438-4b8e-96b2-592d1ea387bf","Type":"ContainerDied","Data":"edff7fd72bac38fdaa2924c59cbf84fd9a074e8c60c5695257deecceacc27ae6"} Oct 30 17:15:04.051307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.050513 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" Oct 30 17:15:04.051496 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.051228 4828 scope.go:122] "RemoveContainer" containerID="1314121c0bebb10cf5e89b39b43f1226d204fad1e8bc2f3f3d1517f883568f2d" Oct 30 17:15:04.055088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.055062 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" event={"ID":"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1","Type":"ContainerStarted","Data":"99e55baae9ae6694d43317f0572e7ebca0b636b0a34d8e21a5f4b34622701d3b"} Oct 30 17:15:04.124113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.119050 4828 scope.go:122] "RemoveContainer" containerID="1314121c0bebb10cf5e89b39b43f1226d204fad1e8bc2f3f3d1517f883568f2d" Oct 30 17:15:04.124113 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:15:04.119467 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1314121c0bebb10cf5e89b39b43f1226d204fad1e8bc2f3f3d1517f883568f2d\": container with ID starting with 1314121c0bebb10cf5e89b39b43f1226d204fad1e8bc2f3f3d1517f883568f2d not found: ID does not exist" containerID="1314121c0bebb10cf5e89b39b43f1226d204fad1e8bc2f3f3d1517f883568f2d" Oct 30 17:15:04.124113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.119505 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1314121c0bebb10cf5e89b39b43f1226d204fad1e8bc2f3f3d1517f883568f2d"} err="failed to get container status \"1314121c0bebb10cf5e89b39b43f1226d204fad1e8bc2f3f3d1517f883568f2d\": rpc error: code = NotFound desc = could not find container \"1314121c0bebb10cf5e89b39b43f1226d204fad1e8bc2f3f3d1517f883568f2d\": container with ID starting with 1314121c0bebb10cf5e89b39b43f1226d204fad1e8bc2f3f3d1517f883568f2d not found: ID does not exist" Oct 30 17:15:04.148812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.148774 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9875cf06-8438-4b8e-96b2-592d1ea387bf-kube-api-access-5c9r4\" (UniqueName: \"kubernetes.io/projected/9875cf06-8438-4b8e-96b2-592d1ea387bf-kube-api-access-5c9r4\") pod \"9875cf06-8438-4b8e-96b2-592d1ea387bf\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " Oct 30 17:15:04.149077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.148838 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-dshm\") pod \"9875cf06-8438-4b8e-96b2-592d1ea387bf\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " Oct 30 17:15:04.149077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.148931 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-tmp-dir\") pod \"9875cf06-8438-4b8e-96b2-592d1ea387bf\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " Oct 30 17:15:04.149077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.148992 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9875cf06-8438-4b8e-96b2-592d1ea387bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/9875cf06-8438-4b8e-96b2-592d1ea387bf-tls-certs\") pod \"9875cf06-8438-4b8e-96b2-592d1ea387bf\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " Oct 30 17:15:04.149077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.149054 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-model-cache\") pod \"9875cf06-8438-4b8e-96b2-592d1ea387bf\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " Oct 30 17:15:04.149310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.149107 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-home\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-home\") pod \"9875cf06-8438-4b8e-96b2-592d1ea387bf\" (UID: \"9875cf06-8438-4b8e-96b2-592d1ea387bf\") " Oct 30 17:15:04.151769 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.151725 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-home" pod "9875cf06-8438-4b8e-96b2-592d1ea387bf" (UID: "9875cf06-8438-4b8e-96b2-592d1ea387bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:04.153284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.153230 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-tmp-dir" pod "9875cf06-8438-4b8e-96b2-592d1ea387bf" (UID: "9875cf06-8438-4b8e-96b2-592d1ea387bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:04.156233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.156203 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9875cf06-8438-4b8e-96b2-592d1ea387bf-kube-api-access-5c9r4" pod "9875cf06-8438-4b8e-96b2-592d1ea387bf" (UID: "9875cf06-8438-4b8e-96b2-592d1ea387bf"). InnerVolumeSpecName "kube-api-access-5c9r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:04.156545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.156440 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-model-cache" pod "9875cf06-8438-4b8e-96b2-592d1ea387bf" (UID: "9875cf06-8438-4b8e-96b2-592d1ea387bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:04.160882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.160859 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt"] Oct 30 17:15:04.163397 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.163369 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-dshm" pod "9875cf06-8438-4b8e-96b2-592d1ea387bf" (UID: "9875cf06-8438-4b8e-96b2-592d1ea387bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:04.163781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.163761 4828 cpu_manager.go:414] "containerMap: removing container" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerName="main" Oct 30 17:15:04.163827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.163785 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerName="main" Oct 30 17:15:04.164087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.164073 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerName="main" Oct 30 17:15:04.165018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.164993 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9875cf06-8438-4b8e-96b2-592d1ea387bf-tls-certs" pod "9875cf06-8438-4b8e-96b2-592d1ea387bf" (UID: "9875cf06-8438-4b8e-96b2-592d1ea387bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:04.212241 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.209477 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt"] Oct 30 17:15:04.212241 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.209618 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" Oct 30 17:15:04.255951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.233538 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5d67ca\"" Oct 30 17:15:04.255951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.234312 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5d67ca-dockercfg-qs6qg\"" Oct 30 17:15:04.255951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.254352 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c9r4\" (UniqueName: \"kubernetes.io/projected/9875cf06-8438-4b8e-96b2-592d1ea387bf-kube-api-access-5c9r4\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:04.255951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.254385 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:04.255951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.254402 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:04.255951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.254416 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9875cf06-8438-4b8e-96b2-592d1ea387bf-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:04.255951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.254429 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:04.255951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.254440 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9875cf06-8438-4b8e-96b2-592d1ea387bf-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:04.266568 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.262840 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:04.359565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.359493 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfn8z\" (UniqueName: \"kubernetes.io/projected/403f830a-d924-4d8f-a787-1e43cd9f6a5c-kube-api-access-pfn8z\") pod \"payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt\" (UID: \"403f830a-d924-4d8f-a787-1e43cd9f6a5c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" Oct 30 17:15:04.359742 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.359625 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/403f830a-d924-4d8f-a787-1e43cd9f6a5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt\" (UID: \"403f830a-d924-4d8f-a787-1e43cd9f6a5c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" Oct 30 17:15:04.504972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.497254 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfn8z\" (UniqueName: \"kubernetes.io/projected/403f830a-d924-4d8f-a787-1e43cd9f6a5c-kube-api-access-pfn8z\") pod \"payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt\" (UID: \"403f830a-d924-4d8f-a787-1e43cd9f6a5c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" Oct 30 17:15:04.504972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.497381 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/403f830a-d924-4d8f-a787-1e43cd9f6a5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt\" (UID: \"403f830a-d924-4d8f-a787-1e43cd9f6a5c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" Oct 30 17:15:04.508922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.505640 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/403f830a-d924-4d8f-a787-1e43cd9f6a5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt\" (UID: \"403f830a-d924-4d8f-a787-1e43cd9f6a5c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" Oct 30 17:15:04.679922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.679267 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfn8z\" (UniqueName: \"kubernetes.io/projected/403f830a-d924-4d8f-a787-1e43cd9f6a5c-kube-api-access-pfn8z\") pod \"payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt\" (UID: \"403f830a-d924-4d8f-a787-1e43cd9f6a5c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" Oct 30 17:15:04.779461 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.759187 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:04.854199 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.854162 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22"] Oct 30 17:15:04.862398 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.862359 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-728ad7/test-model-728ad7-kserve-7bf595ffb5-lkf22"] Oct 30 17:15:04.955947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:04.955871 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" Oct 30 17:15:05.219930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.203493 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv"] Oct 30 17:15:05.244388 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.244355 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:05.285962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.266269 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-metrics-certs\") pod \"maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:05.285962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.282401 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w75gh\" (UniqueName: \"kubernetes.io/projected/8f31a368-af2f-417f-8863-5ab835b7995f-kube-api-access-w75gh\") pod \"maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:05.285962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.282455 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-maas-api-tls\") pod \"maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:05.285962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.282504 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f31a368-af2f-417f-8863-5ab835b7995f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:05.330984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.305294 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-2d01ec-serving-cert\"" Oct 30 17:15:05.358535 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.357993 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:05.439935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.396021 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-metrics-certs\") pod \"maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:05.439935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.396139 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w75gh\" (UniqueName: \"kubernetes.io/projected/8f31a368-af2f-417f-8863-5ab835b7995f-kube-api-access-w75gh\") pod \"maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:05.439935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.396180 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-maas-api-tls\") pod \"maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:05.439935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.396219 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f31a368-af2f-417f-8863-5ab835b7995f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:05.439935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.397333 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f31a368-af2f-417f-8863-5ab835b7995f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:05.499923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.485686 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-metrics-certs\") pod \"maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:05.596996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.590515 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-maas-api-tls\") pod \"maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:05.867611 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.867420 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:05.970409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:05.968945 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w75gh\" (UniqueName: \"kubernetes.io/projected/8f31a368-af2f-417f-8863-5ab835b7995f-kube-api-access-w75gh\") pod \"maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:06.166520 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:06.166439 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv"] Oct 30 17:15:06.166995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:06.166973 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:06.400254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:06.393165 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt"] Oct 30 17:15:06.519945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:06.518946 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9875cf06-8438-4b8e-96b2-592d1ea387bf" path="/var/lib/kubelet/pods/9875cf06-8438-4b8e-96b2-592d1ea387bf/volumes" Oct 30 17:15:06.599943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:06.599871 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv"] Oct 30 17:15:06.601758 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:15:06.601179 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f31a368_af2f_417f_8863_5ab835b7995f.slice/crio-711f8d1a4ee1ac05deb56bd0963df715e415961beca8d61aeba0ee39743f9e7b WatchSource:0}: Error finding container 711f8d1a4ee1ac05deb56bd0963df715e415961beca8d61aeba0ee39743f9e7b: Status 404 returned error can't find the container with id 711f8d1a4ee1ac05deb56bd0963df715e415961beca8d61aeba0ee39743f9e7b Oct 30 17:15:07.362948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:07.362811 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" event={"ID":"403f830a-d924-4d8f-a787-1e43cd9f6a5c","Type":"ContainerStarted","Data":"0510ea6d346522ca30581245cf8cd6fc15d5f3fd72f176d7f61515e7bb700fe2"} Oct 30 17:15:07.362948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:07.362878 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" event={"ID":"403f830a-d924-4d8f-a787-1e43cd9f6a5c","Type":"ContainerStarted","Data":"556af197cc1608823a57ea86144aad184f03b2a2d15432921101eeba44e525b4"} Oct 30 17:15:07.363254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:07.362980 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" Oct 30 17:15:07.366641 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:07.366610 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" event={"ID":"8f31a368-af2f-417f-8863-5ab835b7995f","Type":"ContainerStarted","Data":"d1b2e9066ce1630ef944e2bbedb8537ecee6b43d0154626315df16b8d5d1d1e8"} Oct 30 17:15:07.366786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:07.366664 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" event={"ID":"8f31a368-af2f-417f-8863-5ab835b7995f","Type":"ContainerStarted","Data":"711f8d1a4ee1ac05deb56bd0963df715e415961beca8d61aeba0ee39743f9e7b"} Oct 30 17:15:07.367430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:07.367410 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:07.411265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:07.409664 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" podStartSLOduration=3.409641022 podStartE2EDuration="3.409641022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:07.405678676 +0000 UTC m=+1685.051188857" watchObservedRunningTime="2026-10-30 17:15:07.409641022 +0000 UTC m=+1685.055151204" Oct 30 17:15:07.453018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:07.452960 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" podStartSLOduration=1.992888239 podStartE2EDuration="2.452945157s" totalImagesPullingTime="460.056918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:07.449142543 +0000 UTC m=+1685.094652725" watchObservedRunningTime="2026-10-30 17:15:07.452945157 +0000 UTC m=+1685.098455337" Oct 30 17:15:07.770101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:07.766807 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:07.933724 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:15:07.933683 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:15:08.365094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:08.365049 4828 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:15:08.365358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:08.365126 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:09.457137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:09.457092 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:10.417071 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:10.417025 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" event={"ID":"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1","Type":"ContainerStarted","Data":"5f7bc5a811cf46049fc0770bd8d5a73ec397c0e29fea89bbcd3b7423959c7c56"} Oct 30 17:15:10.460723 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:10.460663 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" podStartSLOduration=4.467146974 podStartE2EDuration="10.460647775s" totalImagesPullingTime="5.993500801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:10.458212578 +0000 UTC m=+1688.103722758" watchObservedRunningTime="2026-10-30 17:15:10.460647775 +0000 UTC m=+1688.106157985" Oct 30 17:15:11.149935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:11.143406 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:11.429391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:11.429308 4828 generic.go:356] "Generic (PLEG): container finished" podID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" containerID="5f7bc5a811cf46049fc0770bd8d5a73ec397c0e29fea89bbcd3b7423959c7c56" exitCode=60 Oct 30 17:15:11.429632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:11.429404 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" event={"ID":"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1","Type":"ContainerDied","Data":"5f7bc5a811cf46049fc0770bd8d5a73ec397c0e29fea89bbcd3b7423959c7c56"} Oct 30 17:15:11.429965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:11.429939 4828 scope.go:122] "RemoveContainer" containerID="5f7bc5a811cf46049fc0770bd8d5a73ec397c0e29fea89bbcd3b7423959c7c56" Oct 30 17:15:11.782023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:11.781988 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:11.963404 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:11.963351 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:11.965777 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:11.965747 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc"] Oct 30 17:15:11.978599 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:11.977329 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.020820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.019364 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc"] Oct 30 17:15:12.072648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.071888 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc599032-c154-45cb-89cd-bebabe6c45e7-istiod-ca-cert\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.072648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.071995 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-podinfo\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.072648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.072085 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.072648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.072113 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-token\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.072648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.072145 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-credential-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.072648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.072329 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9z7m\" (UniqueName: \"kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-kube-api-access-w9z7m\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.072648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.072389 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-certs\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.072648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.072425 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-envoy\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.072648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.072447 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-data\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.174387 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.174354 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.174541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.174398 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-token\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.174541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.174420 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-credential-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.174650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.174597 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9z7m\" (UniqueName: \"kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-kube-api-access-w9z7m\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.174701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.174658 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-certs\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.174753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.174698 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-envoy\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.174753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.174723 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-data\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.174871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.174801 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc599032-c154-45cb-89cd-bebabe6c45e7-istiod-ca-cert\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.174871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.174828 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-podinfo\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.174995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.174881 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-credential-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.174995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.174824 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-socket\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.175157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.175132 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-data\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.175546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.175523 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc599032-c154-45cb-89cd-bebabe6c45e7-istiod-ca-cert\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.175661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.175642 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-certs\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.177339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.177318 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-podinfo\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.177469 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.177453 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-envoy\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.199742 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.199677 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-token\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.199994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.199970 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9z7m\" (UniqueName: \"kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-kube-api-access-w9z7m\") pod \"e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.354973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.354141 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:12.550936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.541232 4828 generic.go:356] "Generic (PLEG): container finished" podID="300e1607-7771-4378-b40a-56cf985f25d5" containerID="6ab43b9e3392c1afa54e24d89e2cdf67e66ade2420a9e7c8bf9588f3e481fd95" exitCode=0 Oct 30 17:15:12.550936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.543582 4828 generic.go:356] "Generic (PLEG): container finished" podID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerID="3921be05aa428d4d55d5f0e1dbc05c2cbfea1131975e2a2d97e29f784c0b47cc" exitCode=0 Oct 30 17:15:12.610153 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.610111 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" event={"ID":"300e1607-7771-4378-b40a-56cf985f25d5","Type":"ContainerDied","Data":"6ab43b9e3392c1afa54e24d89e2cdf67e66ade2420a9e7c8bf9588f3e481fd95"} Oct 30 17:15:12.610420 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.610391 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" event={"ID":"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4","Type":"ContainerDied","Data":"3921be05aa428d4d55d5f0e1dbc05c2cbfea1131975e2a2d97e29f784c0b47cc"} Oct 30 17:15:12.610556 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.610434 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" event={"ID":"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1","Type":"ContainerStarted","Data":"487c867c16bfc4790f2a39d463caad0ba417072f329a85151fe1e96d669a9b24"} Oct 30 17:15:12.612446 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.612422 4828 scope.go:122] "RemoveContainer" containerID="6ab43b9e3392c1afa54e24d89e2cdf67e66ade2420a9e7c8bf9588f3e481fd95" Oct 30 17:15:12.614736 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.614716 4828 scope.go:122] "RemoveContainer" containerID="3921be05aa428d4d55d5f0e1dbc05c2cbfea1131975e2a2d97e29f784c0b47cc" Oct 30 17:15:12.652386 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.652337 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:12.652746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.652722 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:12.652890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.652874 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:12.662831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.657871 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:12.662831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.658098 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:12.662831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.658181 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:12.960358 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:15:12.960312 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc599032_c154_45cb_89cd_bebabe6c45e7.slice/crio-871e37f74be1bdf45b8796604ac176a61fcadc13e3a8158edf63d4c6b32e6d7d WatchSource:0}: Error finding container 871e37f74be1bdf45b8796604ac176a61fcadc13e3a8158edf63d4c6b32e6d7d: Status 404 returned error can't find the container with id 871e37f74be1bdf45b8796604ac176a61fcadc13e3a8158edf63d4c6b32e6d7d Oct 30 17:15:12.981328 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.981265 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:12.981605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.981580 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:12.981782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.981761 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:12.992315 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:12.992258 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc"] Oct 30 17:15:13.283939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:13.282717 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:13.557948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:13.557800 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" event={"ID":"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4","Type":"ContainerStarted","Data":"c65d7a5c9f0e8c38a07aa56cddc0885cbcdedd6a2241c6864c8e48899409b5a4"} Oct 30 17:15:13.562816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:13.562781 4828 generic.go:356] "Generic (PLEG): container finished" podID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" containerID="487c867c16bfc4790f2a39d463caad0ba417072f329a85151fe1e96d669a9b24" exitCode=60 Oct 30 17:15:13.563028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:13.562834 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" event={"ID":"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1","Type":"ContainerDied","Data":"487c867c16bfc4790f2a39d463caad0ba417072f329a85151fe1e96d669a9b24"} Oct 30 17:15:13.563028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:13.562950 4828 scope.go:122] "RemoveContainer" containerID="5f7bc5a811cf46049fc0770bd8d5a73ec397c0e29fea89bbcd3b7423959c7c56" Oct 30 17:15:13.563307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:13.563285 4828 scope.go:122] "RemoveContainer" containerID="487c867c16bfc4790f2a39d463caad0ba417072f329a85151fe1e96d669a9b24" Oct 30 17:15:13.563754 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:15:13.563721 4828 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc_odh-ai-gateway-infra(eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" podUID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" Oct 30 17:15:13.565188 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:13.565149 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" event={"ID":"fc599032-c154-45cb-89cd-bebabe6c45e7","Type":"ContainerStarted","Data":"cb0e1b05afb7f4d6813ded9ae657360a856b4512ce3541d7eba37e2ed13b6abc"} Oct 30 17:15:13.565285 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:13.565200 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" event={"ID":"fc599032-c154-45cb-89cd-bebabe6c45e7","Type":"ContainerStarted","Data":"871e37f74be1bdf45b8796604ac176a61fcadc13e3a8158edf63d4c6b32e6d7d"} Oct 30 17:15:13.567537 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:13.567503 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" event={"ID":"300e1607-7771-4378-b40a-56cf985f25d5","Type":"ContainerStarted","Data":"6569fb90f8f5573fd30307193c2b4d5e08708e90df67e9fa87955f97bbefcbd3"} Oct 30 17:15:13.750739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:13.750679 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" podStartSLOduration=2.750661653 podStartE2EDuration="2.750661653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:13.750345518 +0000 UTC m=+1691.395855699" watchObservedRunningTime="2026-10-30 17:15:13.750661653 +0000 UTC m=+1691.396171829" Oct 30 17:15:13.761106 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:13.761066 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:15:13.761386 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:13.761353 4828 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 17:15:13.761463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:13.761428 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 17:15:14.357926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:14.357223 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:14.357926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:14.357368 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 17:15:14.357926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:14.357416 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" podUID="fc599032-c154-45cb-89cd-bebabe6c45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 17:15:14.412924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:14.412133 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:15:14.412924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:14.412275 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:15:14.412924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:14.412326 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 17:15:14.446926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:14.446128 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:15:15.259120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:15.259075 4828 scope.go:122] "RemoveContainer" containerID="487c867c16bfc4790f2a39d463caad0ba417072f329a85151fe1e96d669a9b24" Oct 30 17:15:15.260163 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:15:15.260127 4828 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc_odh-ai-gateway-infra(eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" podUID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" Oct 30 17:15:15.287021 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:15.276862 4828 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 17:15:15.287021 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:15.276945 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 17:15:15.483486 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:15.483450 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:15.602049 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:15:15.601936 4828 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.116s" Oct 30 17:15:15.771925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:15.770249 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 17:15:15.771925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:15.770335 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" podUID="fc599032-c154-45cb-89cd-bebabe6c45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 17:15:15.846939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:15.806880 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:15:15.846939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:15.807018 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 17:15:15.899982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:15.891767 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:15:15.910066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:15.910024 4828 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 17:15:15.910380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:15.910349 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 17:15:16.052091 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:16.052046 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:16.364926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:16.358304 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 17:15:16.364926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:16.358367 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" podUID="fc599032-c154-45cb-89cd-bebabe6c45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 17:15:16.441927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:16.425947 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:15:16.441927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:16.426011 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 17:15:16.634046 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:16.630371 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 30 17:15:16.763326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:16.763282 4828 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 17:15:16.763618 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:16.763584 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 17:15:17.178511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:17.178465 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" containerID="cri-o://28e469177fca1c3dc43b50cb76a0c906d7976b742ad12fe3a16646b084ce053b" gracePeriod=45 Oct 30 17:15:17.307244 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:17.307194 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:17.407927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:17.371596 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 17:15:17.456032 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:17.455893 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" podUID="fc599032-c154-45cb-89cd-bebabe6c45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 17:15:17.456699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:17.419333 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:15:17.504030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:17.503982 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 17:15:17.774008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:17.771025 4828 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 17:15:17.774008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:17.771082 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 17:15:18.241973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:18.221110 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.145:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:18.270944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:18.270796 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:18.294923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:18.287224 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.145:8000/ready\": dial tcp 10.132.0.145:8000: connect: connection refused" Oct 30 17:15:18.294923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:18.287344 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:15:18.436258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:18.421569 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 17:15:18.436258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:18.421631 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" podUID="fc599032-c154-45cb-89cd-bebabe6c45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 17:15:18.436258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:18.423209 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:18.436258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:18.423827 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:15:18.436258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:18.423867 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 17:15:18.467220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:18.448098 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" Oct 30 17:15:18.806924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:18.759550 4828 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 17:15:18.806924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:18.759605 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 17:15:19.092138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:19.092034 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:19.265585 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:19.265203 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.145:8000/ready\": dial tcp 10.132.0.145:8000: connect: connection refused" Oct 30 17:15:19.358283 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:19.358185 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 17:15:19.358570 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:19.358523 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" podUID="fc599032-c154-45cb-89cd-bebabe6c45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 17:15:19.439422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:19.439380 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 17:15:19.481037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:19.469135 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 17:15:19.871981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:19.842763 4828 patch_prober.go:29] interesting pod/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 30 17:15:19.871981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:19.842942 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 30 17:15:20.066049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.065983 4828 generic.go:356] "Generic (PLEG): container finished" podID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerID="28e469177fca1c3dc43b50cb76a0c906d7976b742ad12fe3a16646b084ce053b" exitCode=0 Oct 30 17:15:20.066049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.066038 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" event={"ID":"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3","Type":"ContainerDied","Data":"28e469177fca1c3dc43b50cb76a0c906d7976b742ad12fe3a16646b084ce053b"} Oct 30 17:15:20.331942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.331747 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:15:20.356462 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.356427 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 17:15:20.356734 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.356702 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" podUID="fc599032-c154-45cb-89cd-bebabe6c45e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 17:15:20.460472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.423612 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-model-cache\") pod \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " Oct 30 17:15:20.460472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.423696 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-home\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-home\") pod \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " Oct 30 17:15:20.460472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.423837 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tls-certs\") pod \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " Oct 30 17:15:20.460472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.424039 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tmp-dir\") pod \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " Oct 30 17:15:20.460472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.424179 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-kube-api-access-vvzd6\" (UniqueName: \"kubernetes.io/projected/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-kube-api-access-vvzd6\") pod \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " Oct 30 17:15:20.460472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.424265 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-dshm\") pod \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\" (UID: \"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3\") " Oct 30 17:15:20.460472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.425357 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tmp-dir" pod "5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" (UID: "5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:20.460472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.425691 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-home" pod "5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" (UID: "5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:20.460472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.429010 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tls-certs" pod "5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" (UID: "5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:20.460472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.429326 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:15:20.460472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.432102 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-model-cache" pod "5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" (UID: "5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:20.460472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.434600 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:15:20.495247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.495203 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-kube-api-access-vvzd6" pod "5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" (UID: "5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3"). InnerVolumeSpecName "kube-api-access-vvzd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:20.526090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.525828 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-dshm" pod "5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" (UID: "5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:20.527772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.527738 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.527988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.527973 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvzd6\" (UniqueName: \"kubernetes.io/projected/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-kube-api-access-vvzd6\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.528106 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.528093 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.528191 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.528181 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.528263 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.528255 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.528336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.528328 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.765213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.765174 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:15:20.765770 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.765747 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:15:20.766606 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.766578 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:15:20.875055 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:20.867926 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:21.087115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:21.087034 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" Oct 30 17:15:21.087766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:21.087714 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" event={"ID":"5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3","Type":"ContainerDied","Data":"26d6675fefc75385485fe51fec9a478ced4d29d9faeed7d6ae1352305c36df49"} Oct 30 17:15:21.087766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:21.087767 4828 scope.go:122] "RemoveContainer" containerID="28e469177fca1c3dc43b50cb76a0c906d7976b742ad12fe3a16646b084ce053b" Oct 30 17:15:21.153675 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:21.153646 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph"] Oct 30 17:15:21.159937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:21.159878 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph"] Oct 30 17:15:21.211776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:21.211736 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-39fecf/test-model-39fecf-kserve-77688688f5-wmtph" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.145:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:21.360683 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:21.360598 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:21.361041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:21.361012 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:21.361708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:21.361689 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:15:22.496003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:22.495969 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" path="/var/lib/kubelet/pods/5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3/volumes" Oct 30 17:15:22.615690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:22.615615 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:22.781380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:22.778569 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:24.446078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:24.446044 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:25.482593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:25.482559 4828 scope.go:122] "RemoveContainer" containerID="487c867c16bfc4790f2a39d463caad0ba417072f329a85151fe1e96d669a9b24" Oct 30 17:15:26.140445 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:26.140355 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" event={"ID":"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1","Type":"ContainerStarted","Data":"8dd912ec8b3456f590e56cd4b9a3ab2cdceb84a0833b5ad61803482c563360d2"} Oct 30 17:15:26.325699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:26.325655 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:27.151669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:27.151634 4828 generic.go:356] "Generic (PLEG): container finished" podID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" containerID="8dd912ec8b3456f590e56cd4b9a3ab2cdceb84a0833b5ad61803482c563360d2" exitCode=60 Oct 30 17:15:27.152137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:27.151708 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" event={"ID":"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1","Type":"ContainerDied","Data":"8dd912ec8b3456f590e56cd4b9a3ab2cdceb84a0833b5ad61803482c563360d2"} Oct 30 17:15:27.152137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:27.151772 4828 scope.go:122] "RemoveContainer" containerID="487c867c16bfc4790f2a39d463caad0ba417072f329a85151fe1e96d669a9b24" Oct 30 17:15:27.152281 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:27.152263 4828 scope.go:122] "RemoveContainer" containerID="8dd912ec8b3456f590e56cd4b9a3ab2cdceb84a0833b5ad61803482c563360d2" Oct 30 17:15:27.152714 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:15:27.152690 4828 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc_odh-ai-gateway-infra(eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" podUID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" Oct 30 17:15:27.195679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:27.195645 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc"] Oct 30 17:15:27.975925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:27.972591 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:28.354937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:28.354880 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" Oct 30 17:15:28.401751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:28.401707 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-kube-api-access-99gr5\" (UniqueName: \"kubernetes.io/projected/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-kube-api-access-99gr5\") pod \"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1\" (UID: \"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1\") " Oct 30 17:15:28.402189 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:28.402153 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-maas-api-service-ca\") pod \"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1\" (UID: \"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1\") " Oct 30 17:15:28.402571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:28.402519 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-maas-api-service-ca" pod "eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" (UID: "eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:28.403680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:28.403655 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-maas-api-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:28.404572 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:28.404544 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-kube-api-access-99gr5" pod "eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" (UID: "eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1"). InnerVolumeSpecName "kube-api-access-99gr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:28.505744 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:28.505708 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99gr5\" (UniqueName: \"kubernetes.io/projected/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1-kube-api-access-99gr5\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:29.173030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:29.172983 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" event={"ID":"eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1","Type":"ContainerDied","Data":"99e55baae9ae6694d43317f0572e7ebca0b636b0a34d8e21a5f4b34622701d3b"} Oct 30 17:15:29.173030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:29.173018 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc" Oct 30 17:15:29.173249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:29.173062 4828 scope.go:122] "RemoveContainer" containerID="8dd912ec8b3456f590e56cd4b9a3ab2cdceb84a0833b5ad61803482c563360d2" Oct 30 17:15:29.236675 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:29.236625 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc"] Oct 30 17:15:29.246416 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:29.246382 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5d8947-29889675-vn7xc"] Oct 30 17:15:29.607426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:29.607322 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:30.499617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:30.499578 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" path="/var/lib/kubelet/pods/eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1/volumes" Oct 30 17:15:31.293539 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:31.293505 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:32.946630 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:32.946586 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:34.345044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:34.345003 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:34.993929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:34.993876 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:35.282635 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:35.282596 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:35.445691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:35.445654 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:35.624608 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:35.624504 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:35.868166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:35.868124 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:37.745563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:37.745520 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:37.915839 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:15:37.915792 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:15:39.594502 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:39.594464 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:41.363764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:41.363725 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:42.183431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:42.183223 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:42.640914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:42.640874 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:43.354734 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:43.354696 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:43.640035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:43.639935 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:44.174819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:44.173519 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:44.514983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:44.492195 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:45.022049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:45.020989 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:45.244935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:45.244192 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:45.474033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:45.430963 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:45.593927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:45.566439 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv"] Oct 30 17:15:45.594427 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:45.594385 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" containerID="cri-o://05afc1b30fb2fd94badffdaa00b9b271ad77979dcbeab81c61fd857c9c5af497" gracePeriod=30 Oct 30 17:15:46.413245 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:46.413197 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:46.631974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:46.631928 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:46.636446 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:46.635950 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:47.444340 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:47.444291 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:47.612405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:47.612371 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:49.987925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:49.983269 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:15:50.099775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:50.099730 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:50.457863 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:50.456930 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:51.063927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.058137 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:51.342489 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.327119 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:15:51.569925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.554931 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:15:51.800100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.800057 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-kube-api-access-brj79\" (UniqueName: \"kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-kube-api-access-brj79\") pod \"3717264a-0933-41fa-9082-221ffa0c6ba5\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " Oct 30 17:15:51.800489 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.800468 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-socket\") pod \"3717264a-0933-41fa-9082-221ffa0c6ba5\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " Oct 30 17:15:51.800635 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.800622 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-podinfo\") pod \"3717264a-0933-41fa-9082-221ffa0c6ba5\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " Oct 30 17:15:51.800749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.800735 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-credential-socket\") pod \"3717264a-0933-41fa-9082-221ffa0c6ba5\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " Oct 30 17:15:51.800892 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.800877 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-certs\") pod \"3717264a-0933-41fa-9082-221ffa0c6ba5\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " Oct 30 17:15:51.801078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.801062 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3717264a-0933-41fa-9082-221ffa0c6ba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3717264a-0933-41fa-9082-221ffa0c6ba5-istiod-ca-cert\") pod \"3717264a-0933-41fa-9082-221ffa0c6ba5\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " Oct 30 17:15:51.801241 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.801227 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-envoy\") pod \"3717264a-0933-41fa-9082-221ffa0c6ba5\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " Oct 30 17:15:51.801408 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.801396 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-data\") pod \"3717264a-0933-41fa-9082-221ffa0c6ba5\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " Oct 30 17:15:51.801525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.801511 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-token\" (UniqueName: \"kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-token\") pod \"3717264a-0933-41fa-9082-221ffa0c6ba5\" (UID: \"3717264a-0933-41fa-9082-221ffa0c6ba5\") " Oct 30 17:15:51.825999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.803500 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-credential-socket" pod "3717264a-0933-41fa-9082-221ffa0c6ba5" (UID: "3717264a-0933-41fa-9082-221ffa0c6ba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:51.825999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.825135 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-socket" pod "3717264a-0933-41fa-9082-221ffa0c6ba5" (UID: "3717264a-0933-41fa-9082-221ffa0c6ba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:51.825999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.825351 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-certs" pod "3717264a-0933-41fa-9082-221ffa0c6ba5" (UID: "3717264a-0933-41fa-9082-221ffa0c6ba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:51.825999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.825690 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3717264a-0933-41fa-9082-221ffa0c6ba5-istiod-ca-cert" pod "3717264a-0933-41fa-9082-221ffa0c6ba5" (UID: "3717264a-0933-41fa-9082-221ffa0c6ba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:51.827320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.827283 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-data" pod "3717264a-0933-41fa-9082-221ffa0c6ba5" (UID: "3717264a-0933-41fa-9082-221ffa0c6ba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:51.847142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.833610 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-kube-api-access-brj79" pod "3717264a-0933-41fa-9082-221ffa0c6ba5" (UID: "3717264a-0933-41fa-9082-221ffa0c6ba5"). InnerVolumeSpecName "kube-api-access-brj79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:51.962580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.916313 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-envoy" pod "3717264a-0933-41fa-9082-221ffa0c6ba5" (UID: "3717264a-0933-41fa-9082-221ffa0c6ba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:51.962580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.916606 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-podinfo" pod "3717264a-0933-41fa-9082-221ffa0c6ba5" (UID: "3717264a-0933-41fa-9082-221ffa0c6ba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:51.993922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.985784 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:51.993922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.985839 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brj79\" (UniqueName: \"kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-kube-api-access-brj79\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:51.993922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.985856 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:51.993922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.985873 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:51.993922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:51.985888 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:52.019977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.013956 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:52.019977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.014012 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3717264a-0933-41fa-9082-221ffa0c6ba5-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:52.019977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.014029 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:52.074783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.074737 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-token" pod "3717264a-0933-41fa-9082-221ffa0c6ba5" (UID: "3717264a-0933-41fa-9082-221ffa0c6ba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:52.106137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.095279 4828 generic.go:356] "Generic (PLEG): container finished" podID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerID="05afc1b30fb2fd94badffdaa00b9b271ad77979dcbeab81c61fd857c9c5af497" exitCode=0 Oct 30 17:15:52.117953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.095310 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" event={"ID":"3717264a-0933-41fa-9082-221ffa0c6ba5","Type":"ContainerDied","Data":"05afc1b30fb2fd94badffdaa00b9b271ad77979dcbeab81c61fd857c9c5af497"} Oct 30 17:15:52.118401 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.118365 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" event={"ID":"3717264a-0933-41fa-9082-221ffa0c6ba5","Type":"ContainerDied","Data":"f4a529f39a48b841735c694b3447678b608580211447e7c167a331ab412559ad"} Oct 30 17:15:52.121998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.116483 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3717264a-0933-41fa-9082-221ffa0c6ba5-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:15:52.122165 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.095397 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv" Oct 30 17:15:52.131885 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.124947 4828 scope.go:122] "RemoveContainer" containerID="05afc1b30fb2fd94badffdaa00b9b271ad77979dcbeab81c61fd857c9c5af497" Oct 30 17:15:52.240738 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.240694 4828 scope.go:122] "RemoveContainer" containerID="861575c602f265de08be93b415495370836f5d8563d0b29fa688e288e560e1cc" Oct 30 17:15:52.331120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.330696 4828 scope.go:122] "RemoveContainer" containerID="05afc1b30fb2fd94badffdaa00b9b271ad77979dcbeab81c61fd857c9c5af497" Oct 30 17:15:52.331785 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:15:52.331429 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05afc1b30fb2fd94badffdaa00b9b271ad77979dcbeab81c61fd857c9c5af497\": container with ID starting with 05afc1b30fb2fd94badffdaa00b9b271ad77979dcbeab81c61fd857c9c5af497 not found: ID does not exist" containerID="05afc1b30fb2fd94badffdaa00b9b271ad77979dcbeab81c61fd857c9c5af497" Oct 30 17:15:52.331785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.331469 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05afc1b30fb2fd94badffdaa00b9b271ad77979dcbeab81c61fd857c9c5af497"} err="failed to get container status \"05afc1b30fb2fd94badffdaa00b9b271ad77979dcbeab81c61fd857c9c5af497\": rpc error: code = NotFound desc = could not find container \"05afc1b30fb2fd94badffdaa00b9b271ad77979dcbeab81c61fd857c9c5af497\": container with ID starting with 05afc1b30fb2fd94badffdaa00b9b271ad77979dcbeab81c61fd857c9c5af497 not found: ID does not exist" Oct 30 17:15:52.331785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.331494 4828 scope.go:122] "RemoveContainer" containerID="861575c602f265de08be93b415495370836f5d8563d0b29fa688e288e560e1cc" Oct 30 17:15:52.332200 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:15:52.332109 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861575c602f265de08be93b415495370836f5d8563d0b29fa688e288e560e1cc\": container with ID starting with 861575c602f265de08be93b415495370836f5d8563d0b29fa688e288e560e1cc not found: ID does not exist" containerID="861575c602f265de08be93b415495370836f5d8563d0b29fa688e288e560e1cc" Oct 30 17:15:52.332200 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.332145 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861575c602f265de08be93b415495370836f5d8563d0b29fa688e288e560e1cc"} err="failed to get container status \"861575c602f265de08be93b415495370836f5d8563d0b29fa688e288e560e1cc\": rpc error: code = NotFound desc = could not find container \"861575c602f265de08be93b415495370836f5d8563d0b29fa688e288e560e1cc\": container with ID starting with 861575c602f265de08be93b415495370836f5d8563d0b29fa688e288e560e1cc not found: ID does not exist" Oct 30 17:15:52.560889 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.560770 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv"] Oct 30 17:15:52.566786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:52.566749 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-728ad7-openshift-default-7959f7dcb5-nd6mv"] Oct 30 17:15:52.661654 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:15:52.661551 4828 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3717264a_0933_41fa_9082_221ffa0c6ba5.slice/crio-f4a529f39a48b841735c694b3447678b608580211447e7c167a331ab412559ad\": RecentStats: unable to find data in memory cache]" Oct 30 17:15:53.385021 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:53.384983 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:54.503922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:54.503794 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" path="/var/lib/kubelet/pods/3717264a-0933-41fa-9082-221ffa0c6ba5/volumes" Oct 30 17:15:54.988814 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:54.984967 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g"] Oct 30 17:15:54.988814 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:54.985601 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" podUID="7beb4126-0039-4932-be23-21d475782c17" containerName="istio-proxy" containerID="cri-o://434e045a242926fe7bba5213580aec59c028e52bd2b1ae7430d8dccb5bd18ce7" gracePeriod=30 Oct 30 17:15:55.027665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:55.027449 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:55.399438 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:55.399118 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:15:55.399438 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:55.399191 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:56.117151 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:56.117105 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk"] Oct 30 17:15:56.158235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:56.158190 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8"] Oct 30 17:15:56.315167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:56.315130 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:56.501099 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:56.498762 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:56.713527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:56.708093 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:57.056738 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.055599 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:57.280794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.279349 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:57.579914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.566847 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:57.652028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.651988 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk"] Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677311 4828 cpu_manager.go:414] "containerMap: removing container" podUID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" containerName="cleanup" Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677350 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" containerName="cleanup" Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677366 4828 cpu_manager.go:414] "containerMap: removing container" podUID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" containerName="cleanup" Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677374 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" containerName="cleanup" Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677387 4828 cpu_manager.go:414] "containerMap: removing container" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677396 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677446 4828 cpu_manager.go:414] "containerMap: removing container" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677454 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677497 4828 cpu_manager.go:414] "containerMap: removing container" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677503 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677815 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677843 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" containerName="cleanup" Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677853 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" containerName="cleanup" Oct 30 17:15:57.750037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.677865 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a2f6b17-a32f-4a17-ba95-f2d5d62be9f3" containerName="main" Oct 30 17:15:57.831310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.825571 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:57.882192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.873199 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:57.882192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.879253 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk"] Oct 30 17:15:57.943357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.937443 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e5ac8f-openshift-default-dockercfg-cgpj4\"" Oct 30 17:15:57.965922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.959017 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1882faef-b475-4456-888e-4ecb07d50831-istio-podinfo\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:57.965922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.959089 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-data\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:57.965922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.959241 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtz4l\" (UniqueName: \"kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-kube-api-access-wtz4l\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:57.965922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.959303 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-credential-socket\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:57.965922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.959370 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1882faef-b475-4456-888e-4ecb07d50831-istiod-ca-cert\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:57.965922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.959480 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-istio-token\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:57.965922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.959545 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-certs\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:57.965922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.959593 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-socket\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:57.965922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:57.959683 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-envoy\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.041174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.041132 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:15:58.062186 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.062102 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-certs\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.062186 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.062184 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-socket\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.062437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.062262 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-envoy\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.062437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.062409 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1882faef-b475-4456-888e-4ecb07d50831-istio-podinfo\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.062554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.062445 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-data\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.062554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.062496 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-certs\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.062661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.062574 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtz4l\" (UniqueName: \"kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-kube-api-access-wtz4l\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.062661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.062611 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-credential-socket\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.062661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.062648 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1882faef-b475-4456-888e-4ecb07d50831-istiod-ca-cert\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.062821 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.062696 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-istio-token\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.062872 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.062825 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-data\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.063506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.063132 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-socket\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.063805 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.063772 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-credential-socket\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.064088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.064054 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1882faef-b475-4456-888e-4ecb07d50831-istiod-ca-cert\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.066486 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.066460 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1882faef-b475-4456-888e-4ecb07d50831-istio-podinfo\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.066789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.066767 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-envoy\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.088302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.088200 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-istio-token\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.091576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.091537 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtz4l\" (UniqueName: \"kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-kube-api-access-wtz4l\") pod \"e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.306243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.306197 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:15:58.721258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.721223 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk"] Oct 30 17:15:58.723509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.723462 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:58.723642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.723540 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:58.723642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:58.723573 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:15:59.199051 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.199006 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q"] Oct 30 17:15:59.199968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.199944 4828 cpu_manager.go:414] "containerMap: removing container" podUID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" containerName="cleanup" Oct 30 17:15:59.199968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.199967 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" containerName="cleanup" Oct 30 17:15:59.200216 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.200202 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="3717264a-0933-41fa-9082-221ffa0c6ba5" containerName="istio-proxy" Oct 30 17:15:59.200280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.200228 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec6ba67-fcac-4b8f-bfc2-ad7dc83786d1" containerName="cleanup" Oct 30 17:15:59.223291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.223254 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:15:59.246974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.244412 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q"] Oct 30 17:15:59.390274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.390237 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx7lp\" (UniqueName: \"kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-kube-api-access-bx7lp\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:15:59.390274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.390274 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:15:59.390882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.390308 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:15:59.390882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.390460 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:15:59.390882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.390491 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:15:59.390882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.390628 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:15:59.390882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.390780 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:15:59.391230 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.390950 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:15:59.391230 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.391027 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:15:59.423454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.408946 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" event={"ID":"1882faef-b475-4456-888e-4ecb07d50831","Type":"ContainerStarted","Data":"97e63113f26c350f1a587a9fde05592303709113bc4c69b451391a463a6d4970"} Oct 30 17:15:59.423454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.409008 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" event={"ID":"1882faef-b475-4456-888e-4ecb07d50831","Type":"ContainerStarted","Data":"6b656e8fe13d3d02b9354fc28413e3419ae747cb25d6fd452ffefa4feb5838b8"} Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.482972 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" podStartSLOduration=2.482951445 podStartE2EDuration="2.482951445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:59.476460351 +0000 UTC m=+1737.121970553" watchObservedRunningTime="2026-10-30 17:15:59.482951445 +0000 UTC m=+1737.128461625" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.492529 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.492645 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.493071 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.493151 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.493264 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.493384 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.493456 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.493580 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx7lp\" (UniqueName: \"kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-kube-api-access-bx7lp\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.493606 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.494280 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-certs\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.494878 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.495254 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istiod-ca-cert\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.495494 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-credential-socket\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.496171 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-data\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.496509 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-podinfo\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.498484 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-envoy\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.525841 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-token\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.526119 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx7lp\" (UniqueName: \"kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-kube-api-access-bx7lp\") pod \"e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:15:59.575502 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.094077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.049658 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:00.283999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.124927 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q"] Oct 30 17:16:00.283999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.125153 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:00.283999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.125243 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:00.283999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.125288 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:00.314203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.314160 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:16:00.330964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.326413 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:16:00.457530 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.457489 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" event={"ID":"4e26d484-11cd-4ef4-a019-6fe388fa0e9d","Type":"ContainerStarted","Data":"dab5bf9b8df1c23d2ec873bd2f621ed18bf94e3b4676c549bea1808dcb4f2057"} Oct 30 17:16:00.459476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.459448 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" event={"ID":"4e26d484-11cd-4ef4-a019-6fe388fa0e9d","Type":"ContainerStarted","Data":"d7bfe4e7947423ae02f966e1ab0b524cc91b48b40c22e51ae16c7098cb0a32e6"} Oct 30 17:16:00.462661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.461296 4828 generic.go:356] "Generic (PLEG): container finished" podID="7beb4126-0039-4932-be23-21d475782c17" containerID="434e045a242926fe7bba5213580aec59c028e52bd2b1ae7430d8dccb5bd18ce7" exitCode=0 Oct 30 17:16:00.462842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.462821 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" event={"ID":"7beb4126-0039-4932-be23-21d475782c17","Type":"ContainerDied","Data":"434e045a242926fe7bba5213580aec59c028e52bd2b1ae7430d8dccb5bd18ce7"} Oct 30 17:16:00.463003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.462985 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" event={"ID":"7beb4126-0039-4932-be23-21d475782c17","Type":"ContainerDied","Data":"71e2ee12b2a7bb43876f0dc983374846be09a15449310aa69fe43ab9acb43543"} Oct 30 17:16:00.463138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.463123 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71e2ee12b2a7bb43876f0dc983374846be09a15449310aa69fe43ab9acb43543" Oct 30 17:16:00.463241 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.463230 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:16:00.464634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.464616 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:16:00.471299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.471268 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 17:16:00.514419 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.514367 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" podStartSLOduration=1.514350429 podStartE2EDuration="1.514350429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:00.508151171 +0000 UTC m=+1738.153661353" watchObservedRunningTime="2026-10-30 17:16:00.514350429 +0000 UTC m=+1738.159860610" Oct 30 17:16:00.614236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.600224 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.642152 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-istio-token\" (UniqueName: \"kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-istio-token\") pod \"7beb4126-0039-4932-be23-21d475782c17\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.642236 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-credential-socket\") pod \"7beb4126-0039-4932-be23-21d475782c17\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.643018 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-data\") pod \"7beb4126-0039-4932-be23-21d475782c17\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.643109 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-socket\") pod \"7beb4126-0039-4932-be23-21d475782c17\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.643264 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7beb4126-0039-4932-be23-21d475782c17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7beb4126-0039-4932-be23-21d475782c17-istio-podinfo\") pod \"7beb4126-0039-4932-be23-21d475782c17\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.643357 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-kube-api-access-h82vc\" (UniqueName: \"kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-kube-api-access-h82vc\") pod \"7beb4126-0039-4932-be23-21d475782c17\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.643487 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-envoy\") pod \"7beb4126-0039-4932-be23-21d475782c17\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.643595 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7beb4126-0039-4932-be23-21d475782c17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7beb4126-0039-4932-be23-21d475782c17-istiod-ca-cert\") pod \"7beb4126-0039-4932-be23-21d475782c17\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.643686 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-certs\") pod \"7beb4126-0039-4932-be23-21d475782c17\" (UID: \"7beb4126-0039-4932-be23-21d475782c17\") " Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.646620 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-certs" pod "7beb4126-0039-4932-be23-21d475782c17" (UID: "7beb4126-0039-4932-be23-21d475782c17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.646814 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-credential-socket" pod "7beb4126-0039-4932-be23-21d475782c17" (UID: "7beb4126-0039-4932-be23-21d475782c17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.647090 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-data" pod "7beb4126-0039-4932-be23-21d475782c17" (UID: "7beb4126-0039-4932-be23-21d475782c17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.647288 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-socket" pod "7beb4126-0039-4932-be23-21d475782c17" (UID: "7beb4126-0039-4932-be23-21d475782c17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.648515 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7beb4126-0039-4932-be23-21d475782c17-istiod-ca-cert" pod "7beb4126-0039-4932-be23-21d475782c17" (UID: "7beb4126-0039-4932-be23-21d475782c17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:00.658235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.656074 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-istio-token" pod "7beb4126-0039-4932-be23-21d475782c17" (UID: "7beb4126-0039-4932-be23-21d475782c17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:00.674420 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.674364 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-kube-api-access-h82vc" pod "7beb4126-0039-4932-be23-21d475782c17" (UID: "7beb4126-0039-4932-be23-21d475782c17"). InnerVolumeSpecName "kube-api-access-h82vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:00.680964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.674751 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7beb4126-0039-4932-be23-21d475782c17-istio-podinfo" pod "7beb4126-0039-4932-be23-21d475782c17" (UID: "7beb4126-0039-4932-be23-21d475782c17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:00.680964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.679336 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-envoy" pod "7beb4126-0039-4932-be23-21d475782c17" (UID: "7beb4126-0039-4932-be23-21d475782c17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:00.757563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.749469 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h82vc\" (UniqueName: \"kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-kube-api-access-h82vc\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.757563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.749531 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.757563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.749547 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7beb4126-0039-4932-be23-21d475782c17-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.757563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.749577 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.757563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.749592 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7beb4126-0039-4932-be23-21d475782c17-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.757563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.749605 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.757563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.749618 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.757563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.749632 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beb4126-0039-4932-be23-21d475782c17-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:00.757563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:00.749670 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7beb4126-0039-4932-be23-21d475782c17-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:01.126929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.119732 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" podUID="99192812-9c04-4cfc-a866-f4f0afee30de" containerName="payload-processing" containerID="cri-o://33863d549bad9df8be49bd335f23e9b1a7d2a12524bc00da84471d5aff662306" gracePeriod=25 Oct 30 17:16:01.158914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.158835 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" podUID="c83f9c48-b4d3-49fb-abc9-36c466cc00f2" containerName="payload-pre-processing" containerID="cri-o://69142c7894feead9cbc4790727c6490886eaaa6edaf18fdf80fa5225779795cb" gracePeriod=25 Oct 30 17:16:01.302680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.302586 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" podUID="c83f9c48-b4d3-49fb-abc9-36c466cc00f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:01.302680 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:16:01.302621 4828 logging.go:55] [core] [Channel #3612 SubChannel #3613]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.133:50052", ServerName: "10.132.0.133:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.133:50052: connect: connection refused" Oct 30 17:16:01.410039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.409397 4828 patch_prober.go:29] interesting pod/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:16:01.410039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.409463 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" podUID="7beb4126-0039-4932-be23-21d475782c17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:01.519328 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.519287 4828 generic.go:356] "Generic (PLEG): container finished" podID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerID="8432fc75d285267ad575baa2b0a8632340a5e625971a1d58567182113bd726ed" exitCode=0 Oct 30 17:16:01.519817 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.519443 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" event={"ID":"0d18657c-89fd-4f95-af19-096ef42f4e18","Type":"ContainerDied","Data":"8432fc75d285267ad575baa2b0a8632340a5e625971a1d58567182113bd726ed"} Oct 30 17:16:01.520049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.520026 4828 scope.go:122] "RemoveContainer" containerID="8432fc75d285267ad575baa2b0a8632340a5e625971a1d58567182113bd726ed" Oct 30 17:16:01.523708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.523675 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:01.523838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.523754 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:01.523838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.523788 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:01.526723 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.526696 4828 generic.go:356] "Generic (PLEG): container finished" podID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerID="9ff64fd8dd9db293aa8e03fbf2fc3ddc9b3204f7350a532bf9f1bd50a20c04d3" exitCode=0 Oct 30 17:16:01.528056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.526923 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" event={"ID":"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a","Type":"ContainerDied","Data":"9ff64fd8dd9db293aa8e03fbf2fc3ddc9b3204f7350a532bf9f1bd50a20c04d3"} Oct 30 17:16:01.528056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.527321 4828 scope.go:122] "RemoveContainer" containerID="9ff64fd8dd9db293aa8e03fbf2fc3ddc9b3204f7350a532bf9f1bd50a20c04d3" Oct 30 17:16:01.528613 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.528591 4828 generic.go:356] "Generic (PLEG): container finished" podID="99192812-9c04-4cfc-a866-f4f0afee30de" containerID="33863d549bad9df8be49bd335f23e9b1a7d2a12524bc00da84471d5aff662306" exitCode=0 Oct 30 17:16:01.528727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.528710 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" event={"ID":"99192812-9c04-4cfc-a866-f4f0afee30de","Type":"ContainerDied","Data":"33863d549bad9df8be49bd335f23e9b1a7d2a12524bc00da84471d5aff662306"} Oct 30 17:16:01.529435 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.529407 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:01.529504 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.529492 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:01.529565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.529541 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:01.531637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.531618 4828 generic.go:356] "Generic (PLEG): container finished" podID="15f58c09-e14a-41e4-a355-c08c7575c648" containerID="3eecf2d5680e61f434ed7d8651f875ace7b03a6030cb64842a3dbf097a10981f" exitCode=0 Oct 30 17:16:01.532356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.532333 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" event={"ID":"15f58c09-e14a-41e4-a355-c08c7575c648","Type":"ContainerDied","Data":"3eecf2d5680e61f434ed7d8651f875ace7b03a6030cb64842a3dbf097a10981f"} Oct 30 17:16:01.532753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.532735 4828 scope.go:122] "RemoveContainer" containerID="3eecf2d5680e61f434ed7d8651f875ace7b03a6030cb64842a3dbf097a10981f" Oct 30 17:16:01.533262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.533245 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g" Oct 30 17:16:01.538613 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.538503 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:01.539097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.538589 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:01.539097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.538765 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:01.624716 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.624684 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 17:16:01.625592 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.625562 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 17:16:01.719281 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.719183 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" Oct 30 17:16:01.876053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.863063 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:16:01.971101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.963690 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99192812-9c04-4cfc-a866-f4f0afee30de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99192812-9c04-4cfc-a866-f4f0afee30de-plugins-config-volume\") pod \"99192812-9c04-4cfc-a866-f4f0afee30de\" (UID: \"99192812-9c04-4cfc-a866-f4f0afee30de\") " Oct 30 17:16:01.971101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.964025 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99192812-9c04-4cfc-a866-f4f0afee30de-kube-api-access-rqk6t\" (UniqueName: \"kubernetes.io/projected/99192812-9c04-4cfc-a866-f4f0afee30de-kube-api-access-rqk6t\") pod \"99192812-9c04-4cfc-a866-f4f0afee30de\" (UID: \"99192812-9c04-4cfc-a866-f4f0afee30de\") " Oct 30 17:16:01.971101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:01.965264 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99192812-9c04-4cfc-a866-f4f0afee30de-plugins-config-volume" pod "99192812-9c04-4cfc-a866-f4f0afee30de" (UID: "99192812-9c04-4cfc-a866-f4f0afee30de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:02.038029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.033606 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:02.038029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.037977 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g"] Oct 30 17:16:02.046516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.046466 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-jmx4g"] Oct 30 17:16:02.068937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.052048 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99192812-9c04-4cfc-a866-f4f0afee30de-kube-api-access-rqk6t" pod "99192812-9c04-4cfc-a866-f4f0afee30de" (UID: "99192812-9c04-4cfc-a866-f4f0afee30de"). InnerVolumeSpecName "kube-api-access-rqk6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:02.116224 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.116176 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqk6t\" (UniqueName: \"kubernetes.io/projected/99192812-9c04-4cfc-a866-f4f0afee30de-kube-api-access-rqk6t\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:02.116443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.116426 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99192812-9c04-4cfc-a866-f4f0afee30de-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:02.258729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.258696 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:02.317342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.317299 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" podUID="99192812-9c04-4cfc-a866-f4f0afee30de" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.133:50052\" within 1s: context deadline exceeded" Oct 30 17:16:02.318301 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:16:02.318008 4828 logging.go:55] [core] [Channel #3612 SubChannel #3613]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.133:50052", ServerName: "10.132.0.133:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.133:50052: operation was canceled" Oct 30 17:16:02.463926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.463572 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:02.615007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.597965 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 17:16:02.615007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.598023 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 17:16:02.757887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.757848 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7beb4126-0039-4932-be23-21d475782c17" path="/var/lib/kubelet/pods/7beb4126-0039-4932-be23-21d475782c17/volumes" Oct 30 17:16:02.759383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.758675 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:16:02.759383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.758725 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" event={"ID":"15f58c09-e14a-41e4-a355-c08c7575c648","Type":"ContainerStarted","Data":"7209be06affa8802f4fe4e841c4390ef0b92b08532918c50da04d2b1b5faf541"} Oct 30 17:16:02.759383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.758757 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" event={"ID":"0d18657c-89fd-4f95-af19-096ef42f4e18","Type":"ContainerStarted","Data":"3542f6d908e1a4ffffae1d8a8b73c1532976292cb19fcae70583f4438485a602"} Oct 30 17:16:02.764188 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.764159 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" event={"ID":"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a","Type":"ContainerStarted","Data":"c21fd047f78720edd77463dee0fa38b866b64bff3e15487a8b18b814982c53c5"} Oct 30 17:16:02.767441 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.767415 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" event={"ID":"99192812-9c04-4cfc-a866-f4f0afee30de","Type":"ContainerDied","Data":"d2f0a02d4def22361ee95836d68e3fbb06823e2c6a55d7c60890fba2a4d22a9b"} Oct 30 17:16:02.767687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.767665 4828 scope.go:122] "RemoveContainer" containerID="33863d549bad9df8be49bd335f23e9b1a7d2a12524bc00da84471d5aff662306" Oct 30 17:16:02.767941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:02.767460 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk" Oct 30 17:16:03.148924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:03.140499 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:16:03.169933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:03.162709 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 17:16:03.169933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:03.162800 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 17:16:03.291988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:03.284246 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk"] Oct 30 17:16:03.383935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:03.382978 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-39fecf-5998c66dd6-8lrgk"] Oct 30 17:16:03.399982 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:16:03.396288 4828 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99192812_9c04_4cfc_a866_f4f0afee30de.slice/crio-d2f0a02d4def22361ee95836d68e3fbb06823e2c6a55d7c60890fba2a4d22a9b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99192812_9c04_4cfc_a866_f4f0afee30de.slice\": RecentStats: unable to find data in memory cache]" Oct 30 17:16:03.554939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:03.553947 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:16:03.554939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:03.554095 4828 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:16:03.554939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:03.554139 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 17:16:03.581926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:03.577798 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 17:16:03.581926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:03.577869 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 17:16:03.620992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:03.616129 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:16:03.620992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:03.616269 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:16:03.620992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:03.616313 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 17:16:04.131926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.131322 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 17:16:04.131926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.131406 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 17:16:04.430983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.364259 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:04.431166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.431000 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:16:04.589956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.585517 4828 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:16:04.589956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.585668 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 17:16:04.589956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.588984 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 17:16:04.823918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.820282 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:16:04.823918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.820363 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 17:16:04.853235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.848084 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:04.881605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.881265 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99192812-9c04-4cfc-a866-f4f0afee30de" path="/var/lib/kubelet/pods/99192812-9c04-4cfc-a866-f4f0afee30de/volumes" Oct 30 17:16:04.895167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.895058 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 17:16:04.895465 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.895434 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 17:16:04.987764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:04.987662 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:16:05.152057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.151119 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 17:16:05.152057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.151178 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 17:16:05.167671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.167623 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt"] Oct 30 17:16:05.171943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.169091 4828 cpu_manager.go:414] "containerMap: removing container" podUID="99192812-9c04-4cfc-a866-f4f0afee30de" containerName="payload-processing" Oct 30 17:16:05.171943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.169120 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99192812-9c04-4cfc-a866-f4f0afee30de" containerName="payload-processing" Oct 30 17:16:05.171943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.169181 4828 cpu_manager.go:414] "containerMap: removing container" podUID="7beb4126-0039-4932-be23-21d475782c17" containerName="istio-proxy" Oct 30 17:16:05.171943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.169189 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7beb4126-0039-4932-be23-21d475782c17" containerName="istio-proxy" Oct 30 17:16:05.171943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.169437 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="99192812-9c04-4cfc-a866-f4f0afee30de" containerName="payload-processing" Oct 30 17:16:05.171943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.169465 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="7beb4126-0039-4932-be23-21d475782c17" containerName="istio-proxy" Oct 30 17:16:05.318946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.302659 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" Oct 30 17:16:05.318946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.305561 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:05.346380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.326810 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e5ac8f\"" Oct 30 17:16:05.373093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.365177 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8tvf\" (UniqueName: \"kubernetes.io/projected/6fa7d177-2a63-46de-beda-a00ca0f153a4-kube-api-access-j8tvf\") pod \"payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt\" (UID: \"6fa7d177-2a63-46de-beda-a00ca0f153a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" Oct 30 17:16:05.373093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.365375 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa7d177-2a63-46de-beda-a00ca0f153a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt\" (UID: \"6fa7d177-2a63-46de-beda-a00ca0f153a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" Oct 30 17:16:05.389965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.389929 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e5ac8f-dockercfg-cx7vw\"" Oct 30 17:16:05.439791 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.439708 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt"] Oct 30 17:16:05.440129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.440099 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:05.442544 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.440755 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:16:05.497781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.496005 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8tvf\" (UniqueName: \"kubernetes.io/projected/6fa7d177-2a63-46de-beda-a00ca0f153a4-kube-api-access-j8tvf\") pod \"payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt\" (UID: \"6fa7d177-2a63-46de-beda-a00ca0f153a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" Oct 30 17:16:05.497781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.496165 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa7d177-2a63-46de-beda-a00ca0f153a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt\" (UID: \"6fa7d177-2a63-46de-beda-a00ca0f153a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" Oct 30 17:16:05.497781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.497435 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa7d177-2a63-46de-beda-a00ca0f153a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt\" (UID: \"6fa7d177-2a63-46de-beda-a00ca0f153a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" Oct 30 17:16:05.612924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.612009 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:05.627823 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.622834 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:16:05.673348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.640876 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:16:05.673348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.641007 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 17:16:05.673348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.641439 4828 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:16:05.673348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.641481 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 17:16:05.673348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.641552 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 17:16:05.673348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.641576 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 17:16:05.673348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.642386 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8tvf\" (UniqueName: \"kubernetes.io/projected/6fa7d177-2a63-46de-beda-a00ca0f153a4-kube-api-access-j8tvf\") pod \"payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt\" (UID: \"6fa7d177-2a63-46de-beda-a00ca0f153a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" Oct 30 17:16:05.737975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:05.737344 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" Oct 30 17:16:06.028549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.027341 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:16:06.065954 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.041031 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:16:06.209977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.175086 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:06.235925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.225234 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 17:16:06.235925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.225392 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 17:16:06.247562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.247520 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:06.346000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.320123 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:06.346000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.320649 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:16:06.425744 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.425650 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:06.434372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.433989 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:06.444270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.443000 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:06.675545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.647283 4828 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:16:06.675545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.647352 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 17:16:06.675545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.647782 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 17:16:06.675545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.647825 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 17:16:06.704573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.676665 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:06.704573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.695987 4828 scope.go:122] "RemoveContainer" containerID="7a4e6a074469f8baeffd911d24cb6754bd22f2deaca54a61e42bcea44d3e79e6" Oct 30 17:16:06.704573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.703773 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:16:06.704573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:06.703825 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 17:16:07.065688 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:07.037635 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:07.435950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:07.410497 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:16:07.554000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:07.512176 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 17:16:07.554000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:07.512252 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 17:16:07.610013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:07.581233 4828 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:16:07.610013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:07.581297 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 17:16:07.678894 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:07.642752 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:16:07.730552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:07.730490 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:07.769132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:07.760141 4828 scope.go:122] "RemoveContainer" containerID="434e045a242926fe7bba5213580aec59c028e52bd2b1ae7430d8dccb5bd18ce7" Oct 30 17:16:07.807607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:07.773414 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 17:16:07.807607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:07.788832 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 17:16:08.144076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.141715 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 17:16:08.144076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.141808 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 17:16:08.223927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.199036 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:16:08.224285 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.224241 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 17:16:08.227259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.227227 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:08.583928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.581928 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:08.670422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.656025 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:08.701926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.692047 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 17:16:08.794133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.794088 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 17:16:08.794859 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.783163 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:08.907243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.883151 4828 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:16:08.907243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.883224 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 17:16:08.907243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.883387 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:16:08.907243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:08.883414 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 17:16:09.283101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.283062 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 17:16:09.283757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.283708 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 17:16:09.285209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.285182 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:09.440822 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.440791 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt"] Oct 30 17:16:09.441349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.441330 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:16:09.610930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.589004 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:16:09.610930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.610190 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:16:09.610930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.610251 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 17:16:09.610930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.610827 4828 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:16:09.610930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.610875 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 17:16:09.660501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.659470 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:09.843923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.843148 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:09.847922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.844842 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:09.847922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.846246 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:09.849130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.849108 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:16:10.001733 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:16:09.994348 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:16:10.001733 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:09.996477 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:10.258929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:10.220622 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 17:16:10.258929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:10.220675 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 17:16:10.319054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:10.296335 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:10.563048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:10.563002 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" event={"ID":"6fa7d177-2a63-46de-beda-a00ca0f153a4","Type":"ContainerStarted","Data":"2a36c10ad219b8633f2510efeec097cbb61aad3fdfab22305845f5ddf16fe30f"} Oct 30 17:16:10.617929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:10.594094 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:16:10.682181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:10.682135 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:10.765093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:10.765059 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:16:10.765350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:10.765325 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 17:16:10.765949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:10.765925 4828 patch_prober.go:29] interesting pod/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:16:10.766118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:10.766099 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 17:16:10.855930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:10.850015 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:11.043874 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.043842 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:16:11.046345 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.046318 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:16:11.046961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.046791 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:16:11.046961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.046859 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:16:11.313682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.313577 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" podUID="c83f9c48-b4d3-49fb-abc9-36c466cc00f2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:11.333749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.327091 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 30 17:16:11.333749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.327151 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 30 17:16:11.562979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.562887 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:16:11.564587 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.564504 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:16:11.578077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.569316 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:16:11.578077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.576070 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" event={"ID":"6fa7d177-2a63-46de-beda-a00ca0f153a4","Type":"ContainerStarted","Data":"43412990b0d51f3c8341e8ec4b434e76ff2ecbc007599ddb0423db53ec9f06bf"} Oct 30 17:16:11.612313 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.612276 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:16:11.613592 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.613564 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:16:11.656843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.656808 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:11.784446 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:11.784394 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" podStartSLOduration=7.7843807 podStartE2EDuration="7.7843807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:11.78135834 +0000 UTC m=+1749.426868521" watchObservedRunningTime="2026-10-30 17:16:11.7843807 +0000 UTC m=+1749.429890879" Oct 30 17:16:12.132729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:12.132699 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:16:12.133180 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:12.133161 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:16:12.133830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:12.133810 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:16:12.586276 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:12.586246 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" Oct 30 17:16:13.280182 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:13.280147 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:13.437281 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:13.437247 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:13.584713 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:13.584613 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:15.394240 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:15.392841 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:17.768996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:17.765429 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:19.445112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:19.445075 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:20.378448 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:20.378410 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:20.848329 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:20.848292 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:21.259511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:21.259469 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:21.299811 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:16:21.299787 4828 logging.go:55] [core] [Channel #3701 SubChannel #3702]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.132:50052", ServerName: "10.132.0.132:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.132:50052: connect: connection refused" Oct 30 17:16:21.501037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:21.494677 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:21.713123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:21.694521 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:21.883507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:21.883460 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" Oct 30 17:16:21.886743 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:21.886705 4828 generic.go:356] "Generic (PLEG): container finished" podID="c83f9c48-b4d3-49fb-abc9-36c466cc00f2" containerID="69142c7894feead9cbc4790727c6490886eaaa6edaf18fdf80fa5225779795cb" exitCode=0 Oct 30 17:16:21.887857 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:21.887825 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" event={"ID":"c83f9c48-b4d3-49fb-abc9-36c466cc00f2","Type":"ContainerDied","Data":"69142c7894feead9cbc4790727c6490886eaaa6edaf18fdf80fa5225779795cb"} Oct 30 17:16:21.887983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:21.887956 4828 scope.go:122] "RemoveContainer" containerID="69142c7894feead9cbc4790727c6490886eaaa6edaf18fdf80fa5225779795cb" Oct 30 17:16:21.888144 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:21.888125 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" Oct 30 17:16:21.939034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:21.939004 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-plugins-config-volume\") pod \"c83f9c48-b4d3-49fb-abc9-36c466cc00f2\" (UID: \"c83f9c48-b4d3-49fb-abc9-36c466cc00f2\") " Oct 30 17:16:21.939258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:21.939123 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-kube-api-access-lxbcf\" (UniqueName: \"kubernetes.io/projected/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-kube-api-access-lxbcf\") pod \"c83f9c48-b4d3-49fb-abc9-36c466cc00f2\" (UID: \"c83f9c48-b4d3-49fb-abc9-36c466cc00f2\") " Oct 30 17:16:21.939525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:21.939495 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-plugins-config-volume" pod "c83f9c48-b4d3-49fb-abc9-36c466cc00f2" (UID: "c83f9c48-b4d3-49fb-abc9-36c466cc00f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:21.939871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:21.939846 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:21.941447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:21.941420 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-kube-api-access-lxbcf" pod "c83f9c48-b4d3-49fb-abc9-36c466cc00f2" (UID: "c83f9c48-b4d3-49fb-abc9-36c466cc00f2"). InnerVolumeSpecName "kube-api-access-lxbcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:22.041822 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:22.041732 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxbcf\" (UniqueName: \"kubernetes.io/projected/c83f9c48-b4d3-49fb-abc9-36c466cc00f2-kube-api-access-lxbcf\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:22.241115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:22.241078 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8"] Oct 30 17:16:22.253304 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:22.253269 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8"] Oct 30 17:16:22.299717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:22.299616 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39fecf-bb77dcb5f-jgkn8" podUID="c83f9c48-b4d3-49fb-abc9-36c466cc00f2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.132:50052\" within 1s: context deadline exceeded" Oct 30 17:16:22.497098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:22.497057 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c83f9c48-b4d3-49fb-abc9-36c466cc00f2" path="/var/lib/kubelet/pods/c83f9c48-b4d3-49fb-abc9-36c466cc00f2/volumes" Oct 30 17:16:22.941363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:22.941323 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:23.599626 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:23.599592 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" Oct 30 17:16:24.512808 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:24.512765 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:25.421508 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:25.421473 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:25.706875 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:25.706785 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:25.914993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:25.914952 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:26.079991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:26.079953 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:26.266175 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:26.266120 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:27.659719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:27.659664 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:27.828791 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:27.828754 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:28.273763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:28.185164 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:30.001794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:30.001752 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:32.371254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:32.371220 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:33.497224 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:33.495699 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:34.641935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:34.640519 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:35.085807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:35.085773 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:35.318717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:35.318677 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:35.447517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:35.447431 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw"] Oct 30 17:16:35.447683 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:35.447640 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" containerID="cri-o://0248b2bec6d1bae209786cc66e392611540a6ed1b4200de12c5b35bab71b5188" gracePeriod=30 Oct 30 17:16:35.573943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:35.569672 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:36.392483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:36.392447 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:36.816248 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:36.816213 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:37.029036 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:37.028994 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:37.256990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:37.248106 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:37.697938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:37.680151 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:37.925290 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:16:37.925253 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:16:40.174352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.174314 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv"] Oct 30 17:16:40.175980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.175952 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c83f9c48-b4d3-49fb-abc9-36c466cc00f2" containerName="payload-pre-processing" Oct 30 17:16:40.176128 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.176113 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83f9c48-b4d3-49fb-abc9-36c466cc00f2" containerName="payload-pre-processing" Oct 30 17:16:40.176437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.176419 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83f9c48-b4d3-49fb-abc9-36c466cc00f2" containerName="payload-pre-processing" Oct 30 17:16:40.180730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.180710 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.201358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.201320 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv"] Oct 30 17:16:40.202194 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.202170 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-82243b-serving-cert\"" Oct 30 17:16:40.285147 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.285109 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:40.293231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.293201 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv2md\" (UniqueName: \"kubernetes.io/projected/7a96b22c-8739-43ea-9b55-5ce228ad886a-kube-api-access-kv2md\") pod \"maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.293450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.293428 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a96b22c-8739-43ea-9b55-5ce228ad886a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.293555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.293471 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.293656 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.293638 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.395097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.395049 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.395297 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.395262 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv2md\" (UniqueName: \"kubernetes.io/projected/7a96b22c-8739-43ea-9b55-5ce228ad886a-kube-api-access-kv2md\") pod \"maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.395297 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.395290 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a96b22c-8739-43ea-9b55-5ce228ad886a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.395422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.395309 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.396171 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.396137 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a96b22c-8739-43ea-9b55-5ce228ad886a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.398043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.398015 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.398236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.398213 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.422124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.421850 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv2md\" (UniqueName: \"kubernetes.io/projected/7a96b22c-8739-43ea-9b55-5ce228ad886a-kube-api-access-kv2md\") pod \"maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.505927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.505536 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:40.506169 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.505962 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:40.667919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.666360 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:40.918117 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.918087 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:16:40.972349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.972314 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv"] Oct 30 17:16:40.973778 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:16:40.973746 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a96b22c_8739_43ea_9b55_5ce228ad886a.slice/crio-c52e2b6b69f393ef735b71b2cebb69d72e550843ea62e0b678ea9704da9cb806 WatchSource:0}: Error finding container c52e2b6b69f393ef735b71b2cebb69d72e550843ea62e0b678ea9704da9cb806: Status 404 returned error can't find the container with id c52e2b6b69f393ef735b71b2cebb69d72e550843ea62e0b678ea9704da9cb806 Oct 30 17:16:40.975274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:40.975251 4828 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:16:41.031015 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.030968 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-certs\") pod \"ba9d4488-f73e-4096-bed3-603700716223\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " Oct 30 17:16:41.031188 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.031148 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba9d4488-f73e-4096-bed3-603700716223-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9d4488-f73e-4096-bed3-603700716223-istio-podinfo\") pod \"ba9d4488-f73e-4096-bed3-603700716223\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " Oct 30 17:16:41.031251 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.031204 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-envoy\") pod \"ba9d4488-f73e-4096-bed3-603700716223\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " Oct 30 17:16:41.031314 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.031279 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba9d4488-f73e-4096-bed3-603700716223-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9d4488-f73e-4096-bed3-603700716223-istiod-ca-cert\") pod \"ba9d4488-f73e-4096-bed3-603700716223\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " Oct 30 17:16:41.031372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.031326 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-certs" pod "ba9d4488-f73e-4096-bed3-603700716223" (UID: "ba9d4488-f73e-4096-bed3-603700716223"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.031372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.031340 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-data\") pod \"ba9d4488-f73e-4096-bed3-603700716223\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " Oct 30 17:16:41.031478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.031409 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-istio-token\" (UniqueName: \"kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-istio-token\") pod \"ba9d4488-f73e-4096-bed3-603700716223\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " Oct 30 17:16:41.031834 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.031664 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba9d4488-f73e-4096-bed3-603700716223-istiod-ca-cert" pod "ba9d4488-f73e-4096-bed3-603700716223" (UID: "ba9d4488-f73e-4096-bed3-603700716223"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:41.031834 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.031682 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-data" pod "ba9d4488-f73e-4096-bed3-603700716223" (UID: "ba9d4488-f73e-4096-bed3-603700716223"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.032000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.031971 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-credential-socket\") pod \"ba9d4488-f73e-4096-bed3-603700716223\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " Oct 30 17:16:41.032083 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.032064 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-socket\") pod \"ba9d4488-f73e-4096-bed3-603700716223\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " Oct 30 17:16:41.032356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.032324 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-credential-socket" pod "ba9d4488-f73e-4096-bed3-603700716223" (UID: "ba9d4488-f73e-4096-bed3-603700716223"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.032356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.032339 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-socket" pod "ba9d4488-f73e-4096-bed3-603700716223" (UID: "ba9d4488-f73e-4096-bed3-603700716223"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.035124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.032418 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-kube-api-access-ljvb4\" (UniqueName: \"kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-kube-api-access-ljvb4\") pod \"ba9d4488-f73e-4096-bed3-603700716223\" (UID: \"ba9d4488-f73e-4096-bed3-603700716223\") " Oct 30 17:16:41.035124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.033246 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.035124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.033267 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba9d4488-f73e-4096-bed3-603700716223-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.035124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.033282 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.035124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.033296 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.035124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.033312 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.035124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.034484 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-envoy" pod "ba9d4488-f73e-4096-bed3-603700716223" (UID: "ba9d4488-f73e-4096-bed3-603700716223"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.035124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.034959 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-kube-api-access-ljvb4" pod "ba9d4488-f73e-4096-bed3-603700716223" (UID: "ba9d4488-f73e-4096-bed3-603700716223"). InnerVolumeSpecName "kube-api-access-ljvb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:41.035124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.035114 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba9d4488-f73e-4096-bed3-603700716223-istio-podinfo" pod "ba9d4488-f73e-4096-bed3-603700716223" (UID: "ba9d4488-f73e-4096-bed3-603700716223"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:16:41.036693 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.036673 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-istio-token" pod "ba9d4488-f73e-4096-bed3-603700716223" (UID: "ba9d4488-f73e-4096-bed3-603700716223"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:41.134761 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.134720 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba9d4488-f73e-4096-bed3-603700716223-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.134761 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.134755 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba9d4488-f73e-4096-bed3-603700716223-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.134761 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.134765 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.135013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.134773 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljvb4\" (UniqueName: \"kubernetes.io/projected/ba9d4488-f73e-4096-bed3-603700716223-kube-api-access-ljvb4\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.259022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.258985 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:41.625946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.625852 4828 generic.go:356] "Generic (PLEG): container finished" podID="ba9d4488-f73e-4096-bed3-603700716223" containerID="0248b2bec6d1bae209786cc66e392611540a6ed1b4200de12c5b35bab71b5188" exitCode=0 Oct 30 17:16:41.626080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.625959 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" event={"ID":"ba9d4488-f73e-4096-bed3-603700716223","Type":"ContainerDied","Data":"0248b2bec6d1bae209786cc66e392611540a6ed1b4200de12c5b35bab71b5188"} Oct 30 17:16:41.626080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.626023 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" event={"ID":"ba9d4488-f73e-4096-bed3-603700716223","Type":"ContainerDied","Data":"2d4092c9e203680e6e7ab0f5addc68b00d8048457e348b02ef1abc9f948ddb7b"} Oct 30 17:16:41.626203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.626056 4828 scope.go:122] "RemoveContainer" containerID="0248b2bec6d1bae209786cc66e392611540a6ed1b4200de12c5b35bab71b5188" Oct 30 17:16:41.626278 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.626262 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw" Oct 30 17:16:41.628757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.628726 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" event={"ID":"7a96b22c-8739-43ea-9b55-5ce228ad886a","Type":"ContainerStarted","Data":"c52e2b6b69f393ef735b71b2cebb69d72e550843ea62e0b678ea9704da9cb806"} Oct 30 17:16:41.638306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.637875 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:41.647460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.647436 4828 scope.go:122] "RemoveContainer" containerID="16027ede312e132adb65d31837091d70f90ed6e202d897927844852178931959" Oct 30 17:16:41.662930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.662895 4828 scope.go:122] "RemoveContainer" containerID="0248b2bec6d1bae209786cc66e392611540a6ed1b4200de12c5b35bab71b5188" Oct 30 17:16:41.663326 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:16:41.663298 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0248b2bec6d1bae209786cc66e392611540a6ed1b4200de12c5b35bab71b5188\": container with ID starting with 0248b2bec6d1bae209786cc66e392611540a6ed1b4200de12c5b35bab71b5188 not found: ID does not exist" containerID="0248b2bec6d1bae209786cc66e392611540a6ed1b4200de12c5b35bab71b5188" Oct 30 17:16:41.663391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.663338 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0248b2bec6d1bae209786cc66e392611540a6ed1b4200de12c5b35bab71b5188"} err="failed to get container status \"0248b2bec6d1bae209786cc66e392611540a6ed1b4200de12c5b35bab71b5188\": rpc error: code = NotFound desc = could not find container \"0248b2bec6d1bae209786cc66e392611540a6ed1b4200de12c5b35bab71b5188\": container with ID starting with 0248b2bec6d1bae209786cc66e392611540a6ed1b4200de12c5b35bab71b5188 not found: ID does not exist" Oct 30 17:16:41.663391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.663363 4828 scope.go:122] "RemoveContainer" containerID="16027ede312e132adb65d31837091d70f90ed6e202d897927844852178931959" Oct 30 17:16:41.663695 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:16:41.663670 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16027ede312e132adb65d31837091d70f90ed6e202d897927844852178931959\": container with ID starting with 16027ede312e132adb65d31837091d70f90ed6e202d897927844852178931959 not found: ID does not exist" containerID="16027ede312e132adb65d31837091d70f90ed6e202d897927844852178931959" Oct 30 17:16:41.663751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.663703 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16027ede312e132adb65d31837091d70f90ed6e202d897927844852178931959"} err="failed to get container status \"16027ede312e132adb65d31837091d70f90ed6e202d897927844852178931959\": rpc error: code = NotFound desc = could not find container \"16027ede312e132adb65d31837091d70f90ed6e202d897927844852178931959\": container with ID starting with 16027ede312e132adb65d31837091d70f90ed6e202d897927844852178931959 not found: ID does not exist" Oct 30 17:16:41.713358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.713321 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw"] Oct 30 17:16:41.759925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.754730 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b2c3fb-openshift-default-64c87f5bf5-rn2rw"] Oct 30 17:16:41.842924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:41.842438 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:42.497157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:42.497118 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba9d4488-f73e-4096-bed3-603700716223" path="/var/lib/kubelet/pods/ba9d4488-f73e-4096-bed3-603700716223/volumes" Oct 30 17:16:42.641560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:42.641520 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" event={"ID":"7a96b22c-8739-43ea-9b55-5ce228ad886a","Type":"ContainerStarted","Data":"4ab918c9074b8180cdb74a39e7e970059a39ca222541113af712e5e6d846ac49"} Oct 30 17:16:42.641818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:42.641788 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:42.699155 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:42.699092 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" podStartSLOduration=2.076094216 podStartE2EDuration="2.69907632s" totalImagesPullingTime="622.982104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:42.698293429 +0000 UTC m=+1780.343803610" watchObservedRunningTime="2026-10-30 17:16:42.69907632 +0000 UTC m=+1780.344586500" Oct 30 17:16:43.016314 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:43.015960 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:43.869888 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:43.869847 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:44.036242 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.036205 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:44.062256 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.062206 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd"] Oct 30 17:16:44.063522 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.063487 4828 cpu_manager.go:414] "containerMap: removing container" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" Oct 30 17:16:44.063522 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.063513 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" Oct 30 17:16:44.063725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.063551 4828 cpu_manager.go:414] "containerMap: removing container" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" Oct 30 17:16:44.063725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.063559 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" Oct 30 17:16:44.063837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.063824 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" Oct 30 17:16:44.069828 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.069804 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" Oct 30 17:16:44.102443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.102401 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd"] Oct 30 17:16:44.105763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.105729 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-42d381-dockercfg-8bvjw\"" Oct 30 17:16:44.117847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.112929 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-42d381\"" Oct 30 17:16:44.123824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.123144 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eb5f6d5-47f2-4516-9a67-769660921f0c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd\" (UID: \"7eb5f6d5-47f2-4516-9a67-769660921f0c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" Oct 30 17:16:44.123824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.123262 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzh5n\" (UniqueName: \"kubernetes.io/projected/7eb5f6d5-47f2-4516-9a67-769660921f0c-kube-api-access-bzh5n\") pod \"payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd\" (UID: \"7eb5f6d5-47f2-4516-9a67-769660921f0c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" Oct 30 17:16:44.226660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.226588 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eb5f6d5-47f2-4516-9a67-769660921f0c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd\" (UID: \"7eb5f6d5-47f2-4516-9a67-769660921f0c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" Oct 30 17:16:44.226891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.226779 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzh5n\" (UniqueName: \"kubernetes.io/projected/7eb5f6d5-47f2-4516-9a67-769660921f0c-kube-api-access-bzh5n\") pod \"payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd\" (UID: \"7eb5f6d5-47f2-4516-9a67-769660921f0c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" Oct 30 17:16:44.227391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.227365 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eb5f6d5-47f2-4516-9a67-769660921f0c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd\" (UID: \"7eb5f6d5-47f2-4516-9a67-769660921f0c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" Oct 30 17:16:44.252123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.252083 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzh5n\" (UniqueName: \"kubernetes.io/projected/7eb5f6d5-47f2-4516-9a67-769660921f0c-kube-api-access-bzh5n\") pod \"payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd\" (UID: \"7eb5f6d5-47f2-4516-9a67-769660921f0c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" Oct 30 17:16:44.412297 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.408215 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" Oct 30 17:16:44.970097 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:16:44.970067 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb5f6d5_47f2_4516_9a67_769660921f0c.slice/crio-be7ff0c89b0a2cd0af9cbc0272ea0e7918cb92a394800e209567a5a132b89fcb WatchSource:0}: Error finding container be7ff0c89b0a2cd0af9cbc0272ea0e7918cb92a394800e209567a5a132b89fcb: Status 404 returned error can't find the container with id be7ff0c89b0a2cd0af9cbc0272ea0e7918cb92a394800e209567a5a132b89fcb Oct 30 17:16:44.970909 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:44.970870 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd"] Oct 30 17:16:45.110261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:45.110230 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:45.684528 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:45.684482 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" event={"ID":"7eb5f6d5-47f2-4516-9a67-769660921f0c","Type":"ContainerStarted","Data":"3b5bc05ccc86264d72b56e0802380e4781ce0efb3a801178fc93c9f4b364dc57"} Oct 30 17:16:45.685511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:45.684546 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" event={"ID":"7eb5f6d5-47f2-4516-9a67-769660921f0c","Type":"ContainerStarted","Data":"be7ff0c89b0a2cd0af9cbc0272ea0e7918cb92a394800e209567a5a132b89fcb"} Oct 30 17:16:45.685511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:45.684586 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" Oct 30 17:16:45.733173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:45.733121 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" podStartSLOduration=1.733107567 podStartE2EDuration="1.733107567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:45.729769059 +0000 UTC m=+1783.375279239" watchObservedRunningTime="2026-10-30 17:16:45.733107567 +0000 UTC m=+1783.378617747" Oct 30 17:16:46.698087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:46.698056 4828 generic.go:356] "Generic (PLEG): container finished" podID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerID="daa4cf24bfd00b221e1daa64ff4584e2ff59118ee1b77d95d2572060caf4bac2" exitCode=0 Oct 30 17:16:46.698499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:46.698133 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" event={"ID":"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0","Type":"ContainerDied","Data":"daa4cf24bfd00b221e1daa64ff4584e2ff59118ee1b77d95d2572060caf4bac2"} Oct 30 17:16:46.698855 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:46.698836 4828 scope.go:122] "RemoveContainer" containerID="daa4cf24bfd00b221e1daa64ff4584e2ff59118ee1b77d95d2572060caf4bac2" Oct 30 17:16:46.700681 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:46.700648 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:46.700802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:46.700737 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:46.700802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:46.700784 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:16:47.088597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:47.088551 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:47.715507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:47.715468 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" event={"ID":"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0","Type":"ContainerStarted","Data":"4aa5f80314716a0898827446f8308b5146ccd8bb04b41aee5528cad7f5b90dc7"} Oct 30 17:16:48.649298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:48.647301 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:16:48.649298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:48.648378 4828 patch_prober.go:29] interesting pod/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 17:16:48.649298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:48.648441 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 17:16:48.656552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:48.656521 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:16:48.709363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:48.709129 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:49.626055 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:49.626022 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:49.643470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:49.643430 4828 patch_prober.go:29] interesting pod/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 17:16:49.643658 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:49.643500 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 17:16:49.798925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:49.796512 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:49.967500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:49.967399 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:50.329387 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:50.329343 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:50.450472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:50.450435 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:16:50.643445 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:50.643352 4828 patch_prober.go:29] interesting pod/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 17:16:50.643445 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:50.643413 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 17:16:51.643477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:51.643441 4828 patch_prober.go:29] interesting pod/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 17:16:51.644033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:51.643491 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 17:16:52.026203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:52.026160 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:52.208881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:52.208782 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:52.643403 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:52.643364 4828 patch_prober.go:29] interesting pod/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 17:16:52.643570 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:52.643420 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 17:16:53.649001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:53.648965 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:16:53.650377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:53.650353 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:16:53.904742 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:53.904621 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:54.078797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:54.078762 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:54.256952 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:54.256841 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:55.946643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:55.946602 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:56.702329 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:56.702299 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" Oct 30 17:16:57.220808 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:57.220773 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:57.776963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:57.776869 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:58.019415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:58.019358 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:58.240866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:58.240780 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:58.378020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:58.377982 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh"] Oct 30 17:16:58.378269 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:58.378245 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" podUID="fe67cd22-adb6-4549-8a92-7a86b91cd1ac" containerName="istio-proxy" containerID="cri-o://84311205c96841b56b7559db031452672b43058c252846bbdea84b63fd1e2d58" gracePeriod=30 Oct 30 17:16:58.443854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:58.443814 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:16:59.254141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:16:59.254108 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:00.612996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:00.612953 4828 patch_prober.go:29] interesting pod/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:17:00.613433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:00.613036 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" podUID="fe67cd22-adb6-4549-8a92-7a86b91cd1ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:17:00.711182 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:00.711144 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:01.554150 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:01.554106 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:03.755364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.754785 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:03.758354 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.758320 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:17:03.812704 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.812660 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-certs\") pod \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " Oct 30 17:17:03.812922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.812801 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-envoy\") pod \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " Oct 30 17:17:03.812922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.812842 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-token\" (UniqueName: \"kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-token\") pod \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " Oct 30 17:17:03.812922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.812918 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-podinfo\") pod \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " Oct 30 17:17:03.813102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.812944 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istiod-ca-cert\") pod \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " Oct 30 17:17:03.813102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.813044 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-data\") pod \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " Oct 30 17:17:03.813102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.813080 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-socket\") pod \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " Oct 30 17:17:03.813259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.813151 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-kube-api-access-hx674\" (UniqueName: \"kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-kube-api-access-hx674\") pod \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " Oct 30 17:17:03.813362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.813340 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-credential-socket\") pod \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\" (UID: \"fe67cd22-adb6-4549-8a92-7a86b91cd1ac\") " Oct 30 17:17:03.813834 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.813803 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-data" pod "fe67cd22-adb6-4549-8a92-7a86b91cd1ac" (UID: "fe67cd22-adb6-4549-8a92-7a86b91cd1ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:03.814045 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.814019 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istiod-ca-cert" pod "fe67cd22-adb6-4549-8a92-7a86b91cd1ac" (UID: "fe67cd22-adb6-4549-8a92-7a86b91cd1ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:03.814127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.814053 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-credential-socket" pod "fe67cd22-adb6-4549-8a92-7a86b91cd1ac" (UID: "fe67cd22-adb6-4549-8a92-7a86b91cd1ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:03.814186 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.814114 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-socket" pod "fe67cd22-adb6-4549-8a92-7a86b91cd1ac" (UID: "fe67cd22-adb6-4549-8a92-7a86b91cd1ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:03.814334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.814297 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-certs" pod "fe67cd22-adb6-4549-8a92-7a86b91cd1ac" (UID: "fe67cd22-adb6-4549-8a92-7a86b91cd1ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:03.814334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.814318 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:17:03.814484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.814345 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:17:03.814484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.814359 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:17:03.814484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.814373 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:17:03.816836 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.815675 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-podinfo" pod "fe67cd22-adb6-4549-8a92-7a86b91cd1ac" (UID: "fe67cd22-adb6-4549-8a92-7a86b91cd1ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:03.819260 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.817250 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-envoy" pod "fe67cd22-adb6-4549-8a92-7a86b91cd1ac" (UID: "fe67cd22-adb6-4549-8a92-7a86b91cd1ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:03.820137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.820107 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-token" pod "fe67cd22-adb6-4549-8a92-7a86b91cd1ac" (UID: "fe67cd22-adb6-4549-8a92-7a86b91cd1ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:03.837444 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.837402 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-kube-api-access-hx674" pod "fe67cd22-adb6-4549-8a92-7a86b91cd1ac" (UID: "fe67cd22-adb6-4549-8a92-7a86b91cd1ac"). InnerVolumeSpecName "kube-api-access-hx674". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:03.915625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.915550 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:17:03.915625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.915576 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:17:03.915625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.915589 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:17:03.915625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.915603 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:17:03.915625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.915613 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx674\" (UniqueName: \"kubernetes.io/projected/fe67cd22-adb6-4549-8a92-7a86b91cd1ac-kube-api-access-hx674\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:17:03.958891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.958858 4828 generic.go:356] "Generic (PLEG): container finished" podID="fe67cd22-adb6-4549-8a92-7a86b91cd1ac" containerID="84311205c96841b56b7559db031452672b43058c252846bbdea84b63fd1e2d58" exitCode=0 Oct 30 17:17:03.958891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.958913 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" event={"ID":"fe67cd22-adb6-4549-8a92-7a86b91cd1ac","Type":"ContainerDied","Data":"84311205c96841b56b7559db031452672b43058c252846bbdea84b63fd1e2d58"} Oct 30 17:17:03.958891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.958924 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" Oct 30 17:17:03.959216 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.958954 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh" event={"ID":"fe67cd22-adb6-4549-8a92-7a86b91cd1ac","Type":"ContainerDied","Data":"ce028d65855e38944ede6ab17614e824d55ccdb3f4c2b5edc716b2560087ed2f"} Oct 30 17:17:03.959216 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.958976 4828 scope.go:122] "RemoveContainer" containerID="84311205c96841b56b7559db031452672b43058c252846bbdea84b63fd1e2d58" Oct 30 17:17:03.976377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.974658 4828 scope.go:122] "RemoveContainer" containerID="84311205c96841b56b7559db031452672b43058c252846bbdea84b63fd1e2d58" Oct 30 17:17:03.976377 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:17:03.976083 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84311205c96841b56b7559db031452672b43058c252846bbdea84b63fd1e2d58\": container with ID starting with 84311205c96841b56b7559db031452672b43058c252846bbdea84b63fd1e2d58 not found: ID does not exist" containerID="84311205c96841b56b7559db031452672b43058c252846bbdea84b63fd1e2d58" Oct 30 17:17:03.976377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:03.976125 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84311205c96841b56b7559db031452672b43058c252846bbdea84b63fd1e2d58"} err="failed to get container status \"84311205c96841b56b7559db031452672b43058c252846bbdea84b63fd1e2d58\": rpc error: code = NotFound desc = could not find container \"84311205c96841b56b7559db031452672b43058c252846bbdea84b63fd1e2d58\": container with ID starting with 84311205c96841b56b7559db031452672b43058c252846bbdea84b63fd1e2d58 not found: ID does not exist" Oct 30 17:17:04.018437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:04.018390 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh"] Oct 30 17:17:04.032253 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:04.032208 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-39fecf-openshift-default-f77fc7989-lncrh"] Oct 30 17:17:04.134956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:04.132156 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:04.307617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:04.304102 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:04.552923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:04.552525 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe67cd22-adb6-4549-8a92-7a86b91cd1ac" path="/var/lib/kubelet/pods/fe67cd22-adb6-4549-8a92-7a86b91cd1ac/volumes" Oct 30 17:17:05.424922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:05.424874 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:17:05.431231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:05.431203 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:17:05.815942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:05.812487 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:05.966401 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:05.966367 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:06.118647 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.118553 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:06.466790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.466667 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw"] Oct 30 17:17:06.468502 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.468437 4828 cpu_manager.go:414] "containerMap: removing container" podUID="fe67cd22-adb6-4549-8a92-7a86b91cd1ac" containerName="istio-proxy" Oct 30 17:17:06.468650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.468636 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe67cd22-adb6-4549-8a92-7a86b91cd1ac" containerName="istio-proxy" Oct 30 17:17:06.469034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.469018 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9d4488-f73e-4096-bed3-603700716223" containerName="istio-proxy" Oct 30 17:17:06.469159 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.469150 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe67cd22-adb6-4549-8a92-7a86b91cd1ac" containerName="istio-proxy" Oct 30 17:17:06.526518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.526490 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" Oct 30 17:17:06.545867 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.545843 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-987dfc\"" Oct 30 17:17:06.558959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.558924 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e707eb-e7e5-4464-863d-89f0e71fe2a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw\" (UID: \"68e707eb-e7e5-4464-863d-89f0e71fe2a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" Oct 30 17:17:06.559130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.558981 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj8fs\" (UniqueName: \"kubernetes.io/projected/68e707eb-e7e5-4464-863d-89f0e71fe2a5-kube-api-access-hj8fs\") pod \"payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw\" (UID: \"68e707eb-e7e5-4464-863d-89f0e71fe2a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" Oct 30 17:17:06.563405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.563378 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw"] Oct 30 17:17:06.582588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.582559 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-987dfc-dockercfg-f45hh\"" Oct 30 17:17:06.660298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.660249 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e707eb-e7e5-4464-863d-89f0e71fe2a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw\" (UID: \"68e707eb-e7e5-4464-863d-89f0e71fe2a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" Oct 30 17:17:06.660298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.660305 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj8fs\" (UniqueName: \"kubernetes.io/projected/68e707eb-e7e5-4464-863d-89f0e71fe2a5-kube-api-access-hj8fs\") pod \"payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw\" (UID: \"68e707eb-e7e5-4464-863d-89f0e71fe2a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" Oct 30 17:17:06.661057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.661035 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e707eb-e7e5-4464-863d-89f0e71fe2a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw\" (UID: \"68e707eb-e7e5-4464-863d-89f0e71fe2a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" Oct 30 17:17:06.686766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.686733 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj8fs\" (UniqueName: \"kubernetes.io/projected/68e707eb-e7e5-4464-863d-89f0e71fe2a5-kube-api-access-hj8fs\") pod \"payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw\" (UID: \"68e707eb-e7e5-4464-863d-89f0e71fe2a5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" Oct 30 17:17:06.862746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:06.862709 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" Oct 30 17:17:07.193222 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:17:07.193175 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e707eb_e7e5_4464_863d_89f0e71fe2a5.slice/crio-759dee3b64d49747d81edfd97271054be66e2e4b7d7252f86276b4652b6907b5 WatchSource:0}: Error finding container 759dee3b64d49747d81edfd97271054be66e2e4b7d7252f86276b4652b6907b5: Status 404 returned error can't find the container with id 759dee3b64d49747d81edfd97271054be66e2e4b7d7252f86276b4652b6907b5 Oct 30 17:17:07.196625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:07.196598 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw"] Oct 30 17:17:07.546137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:07.546102 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:07.701766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:07.701728 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:07.937717 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:17:07.937622 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:17:08.001759 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:08.001714 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" event={"ID":"68e707eb-e7e5-4464-863d-89f0e71fe2a5","Type":"ContainerStarted","Data":"46b68ab8b5e66d80341a9a9c6f63ef94cc8e0f538f6025bc5ba1ba0b880bdb6e"} Oct 30 17:17:08.001974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:08.001780 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" event={"ID":"68e707eb-e7e5-4464-863d-89f0e71fe2a5","Type":"ContainerStarted","Data":"759dee3b64d49747d81edfd97271054be66e2e4b7d7252f86276b4652b6907b5"} Oct 30 17:17:08.001974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:08.001837 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" Oct 30 17:17:08.046238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:08.046182 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" podStartSLOduration=2.046167126 podStartE2EDuration="2.046167126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:08.044146314 +0000 UTC m=+1805.689656494" watchObservedRunningTime="2026-10-30 17:17:08.046167126 +0000 UTC m=+1805.691677307" Oct 30 17:17:09.317707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:09.308368 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:09.661826 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:09.661671 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:09.825818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:09.825784 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:11.203923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:11.193189 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:12.735626 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:12.735589 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:13.550146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.550109 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5"] Oct 30 17:17:13.559399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.559084 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.580043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.579144 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f42baf\"/\"kube-root-ca.crt\"" Oct 30 17:17:13.584015 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.581924 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f42baf\"/\"test-model-f42baf-kserve-self-signed-certs\"" Oct 30 17:17:13.584015 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.582058 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f42baf\"/\"default-dockercfg-j2h8x\"" Oct 30 17:17:13.584015 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.582114 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f42baf\"/\"openshift-service-ca.crt\"" Oct 30 17:17:13.601985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.600601 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5"] Oct 30 17:17:13.638299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.638259 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-home\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.638504 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.638413 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14ba43d-a967-4948-9b5e-9258ed521f08-tls-certs\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.638504 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.638495 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-model-cache\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.638649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.638568 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv2zr\" (UniqueName: \"kubernetes.io/projected/e14ba43d-a967-4948-9b5e-9258ed521f08-kube-api-access-rv2zr\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.638711 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.638670 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-dshm\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.638778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.638721 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-tmp-dir\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.656137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.656101 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:13.740295 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.740261 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv2zr\" (UniqueName: \"kubernetes.io/projected/e14ba43d-a967-4948-9b5e-9258ed521f08-kube-api-access-rv2zr\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.740714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.740359 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-dshm\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.740714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.740407 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-tmp-dir\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.740714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.740468 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-home\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.740714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.740599 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14ba43d-a967-4948-9b5e-9258ed521f08-tls-certs\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.740714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.740671 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-model-cache\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.741026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.740851 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-tmp-dir\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.741026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.740976 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-home\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.741142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.741123 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-model-cache\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.742599 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.742572 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-dshm\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.743041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.743023 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14ba43d-a967-4948-9b5e-9258ed521f08-tls-certs\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.764777 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.764745 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv2zr\" (UniqueName: \"kubernetes.io/projected/e14ba43d-a967-4948-9b5e-9258ed521f08-kube-api-access-rv2zr\") pod \"test-model-f42baf-kserve-576f78b8df-65ws5\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:13.910986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:13.910880 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:14.345293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:14.344349 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5"] Oct 30 17:17:14.419776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:14.419740 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:14.593699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:14.592732 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:15.072401 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:15.072368 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" event={"ID":"e14ba43d-a967-4948-9b5e-9258ed521f08","Type":"ContainerStarted","Data":"eacd7f3e6563deda5b4f9776fb6d437be6e4d8fc25ffda68be94d7f2057596e3"} Oct 30 17:17:15.072916 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:15.072416 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" event={"ID":"e14ba43d-a967-4948-9b5e-9258ed521f08","Type":"ContainerStarted","Data":"bceb46c183563be2f5e2a6136a1f21556325750132afe37b162e77cb3ca46f4f"} Oct 30 17:17:15.119553 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:15.119492 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" podStartSLOduration=2.119477342 podStartE2EDuration="2.119477342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:15.116153551 +0000 UTC m=+1812.761663731" watchObservedRunningTime="2026-10-30 17:17:15.119477342 +0000 UTC m=+1812.764987556" Oct 30 17:17:15.912273 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:15.912239 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:15.927312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:15.927256 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:16.877035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:16.876997 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:18.665435 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:18.665357 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:18.999100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:18.999008 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:19.025078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:19.025048 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" Oct 30 17:17:19.205130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:19.205095 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:20.504927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:20.495970 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:22.331149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:22.330983 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:23.911983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:23.911943 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:23.925079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:23.925051 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:23.966164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:23.966093 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:24.256681 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:24.256593 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:17:25.779082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:25.779043 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:27.735820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:27.735775 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:27.910921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:27.907479 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:29.654351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:29.654271 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:31.279559 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:31.279519 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:33.365302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:33.365262 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:34.780407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:34.780371 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:35.173111 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:35.173018 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:36.861278 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:36.861231 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:37.612282 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:37.612242 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:38.015024 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:17:38.014991 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:17:38.742925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:38.737615 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:40.446499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:40.446460 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:42.181574 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:42.181536 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:43.970599 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:43.964928 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:44.610368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:44.610324 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:45.667188 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:45.667150 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:47.519007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:47.516126 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:49.316573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:49.316533 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:49.474923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:49.471542 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:52.331350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:52.331254 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:54.488890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:54.488845 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:56.311924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:56.310623 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:57.397925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:57.390760 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:58.094976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:58.088588 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:58.316599 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:58.316561 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:58.467154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:58.467057 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:17:59.985945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:17:59.978501 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:01.503087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:01.503045 4828 patch_prober.go:29] interesting pod/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:18:01.503480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:01.503131 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:18:01.667691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:01.667600 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:02.822833 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:02.822802 4828 generic.go:356] "Generic (PLEG): container finished" podID="1882faef-b475-4456-888e-4ecb07d50831" containerID="97e63113f26c350f1a587a9fde05592303709113bc4c69b451391a463a6d4970" exitCode=0 Oct 30 17:18:02.823346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:02.822845 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" event={"ID":"1882faef-b475-4456-888e-4ecb07d50831","Type":"ContainerDied","Data":"97e63113f26c350f1a587a9fde05592303709113bc4c69b451391a463a6d4970"} Oct 30 17:18:02.823346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:02.823205 4828 scope.go:122] "RemoveContainer" containerID="97e63113f26c350f1a587a9fde05592303709113bc4c69b451391a463a6d4970" Oct 30 17:18:02.824949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:02.824917 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:02.825028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:02.825008 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:02.825081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:02.825068 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:03.351938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:03.350462 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:03.833157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:03.833122 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" event={"ID":"1882faef-b475-4456-888e-4ecb07d50831","Type":"ContainerStarted","Data":"26d5a9c7ebd6a4d967136327a9842a37d7db93bab9635b9eb876442ad6792c47"} Oct 30 17:18:04.307228 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:04.307181 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:18:04.307577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:04.307551 4828 patch_prober.go:29] interesting pod/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:18:04.307682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:04.307613 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 17:18:05.312922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:05.312471 4828 patch_prober.go:29] interesting pod/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:18:05.312922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:05.312555 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 17:18:05.512783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:05.512747 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:06.307156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:06.307112 4828 patch_prober.go:29] interesting pod/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:18:06.307459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:06.307186 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 17:18:07.306973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:07.306880 4828 patch_prober.go:29] interesting pod/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:18:07.307487 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:07.307004 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 17:18:07.641972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:07.637978 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:07.928444 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:18:07.928327 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:18:07.978537 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:07.978496 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:08.307392 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:08.307360 4828 patch_prober.go:29] interesting pod/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:18:08.308423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:08.308397 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 17:18:09.307764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:09.307715 4828 patch_prober.go:29] interesting pod/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 17:18:09.308335 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:09.307790 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 17:18:09.498929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:09.487997 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:10.311722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.311682 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:18:10.312208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.312188 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:18:10.313223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.313196 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:18:10.691998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.691892 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx"] Oct 30 17:18:10.697992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.697964 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.714715 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.714297 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5d67ca-openshift-default-dockercfg-z2w2w\"" Oct 30 17:18:10.734354 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.734324 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx"] Oct 30 17:18:10.825140 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.825104 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-token\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.825346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.825206 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-envoy\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.825346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.825238 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-certs\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.825346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.825344 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-podinfo\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.825525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.825389 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-data\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.825525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.825407 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istiod-ca-cert\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.825525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.825474 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-credential-socket\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.825698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.825550 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-socket\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.825698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.825659 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkwqt\" (UniqueName: \"kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-kube-api-access-pkwqt\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.927279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.927243 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-socket\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.927458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.927391 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkwqt\" (UniqueName: \"kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-kube-api-access-pkwqt\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.927516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.927482 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-token\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.927568 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.927548 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-envoy\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.927631 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.927575 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-certs\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.927702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.927680 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-podinfo\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.928047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.927728 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-socket\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.928047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.928032 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-data\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.928247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.928065 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istiod-ca-cert\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.928247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.928096 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-credential-socket\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.928861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.928405 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-certs\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.928861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.928581 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-credential-socket\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.928861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.928819 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-data\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.929118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.928985 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istiod-ca-cert\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.930243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.930218 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-envoy\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.934992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.934965 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-podinfo\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.952132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.952060 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-token\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:10.952275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:10.952153 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkwqt\" (UniqueName: \"kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-kube-api-access-pkwqt\") pod \"e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:11.023777 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.023742 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:11.233069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.232851 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:11.358083 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:18:11.358053 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d4cafe_048c_46cf_98dc_381d6eff4d57.slice/crio-4785c5dc1265cb6503630f6132fad3789604a2a3ae1979a05e0ed4c71021bba1 WatchSource:0}: Error finding container 4785c5dc1265cb6503630f6132fad3789604a2a3ae1979a05e0ed4c71021bba1: Status 404 returned error can't find the container with id 4785c5dc1265cb6503630f6132fad3789604a2a3ae1979a05e0ed4c71021bba1 Oct 30 17:18:11.358497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.358473 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx"] Oct 30 17:18:11.361040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.361006 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:11.361154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.361082 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:11.361154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.361128 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:18:11.577976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.577891 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:11.671976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.671939 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22"] Oct 30 17:18:11.684932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.678490 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.699643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.698859 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2d01ec\"/\"test-model-2d01ec-kserve-self-signed-certs\"" Oct 30 17:18:11.699643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.698861 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2d01ec\"/\"openshift-service-ca.crt\"" Oct 30 17:18:11.699643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.698966 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2d01ec\"/\"kube-root-ca.crt\"" Oct 30 17:18:11.709411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.709148 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22"] Oct 30 17:18:11.712044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.710303 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2d01ec\"/\"default-dockercfg-n2945\"" Oct 30 17:18:11.758366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.758304 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-home\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.758572 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.758478 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-dshm\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.758653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.758607 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe970bc-4f76-41e9-93c1-f18b36259ae3-tls-certs\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.758653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.758646 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-model-cache\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.759670 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.759027 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-tmp-dir\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.759670 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.759153 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp56z\" (UniqueName: \"kubernetes.io/projected/afe970bc-4f76-41e9-93c1-f18b36259ae3-kube-api-access-lp56z\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.760681 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.760653 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:11.860824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.860725 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-tmp-dir\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.861050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.860820 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp56z\" (UniqueName: \"kubernetes.io/projected/afe970bc-4f76-41e9-93c1-f18b36259ae3-kube-api-access-lp56z\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.861050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.860963 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-home\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.861050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.861007 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-dshm\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.861208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.861117 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe970bc-4f76-41e9-93c1-f18b36259ae3-tls-certs\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.861208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.861153 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-model-cache\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.861331 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.861285 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-tmp-dir\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.861443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.861401 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-home\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.861585 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.861556 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-model-cache\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.863460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.863436 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-dshm\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.863684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.863667 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe970bc-4f76-41e9-93c1-f18b36259ae3-tls-certs\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:11.914110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:11.908997 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp56z\" (UniqueName: \"kubernetes.io/projected/afe970bc-4f76-41e9-93c1-f18b36259ae3-kube-api-access-lp56z\") pod \"test-model-2d01ec-kserve-9b455ff7-hxq22\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:12.023923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:12.006554 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" event={"ID":"c1d4cafe-048c-46cf-98dc-381d6eff4d57","Type":"ContainerStarted","Data":"b64a3bab6cb39621d145673320a19f87535b008627156f884f84b92b71e17cfb"} Oct 30 17:18:12.023923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:12.006630 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" event={"ID":"c1d4cafe-048c-46cf-98dc-381d6eff4d57","Type":"ContainerStarted","Data":"4785c5dc1265cb6503630f6132fad3789604a2a3ae1979a05e0ed4c71021bba1"} Oct 30 17:18:12.026573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:12.025964 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:12.026573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:12.026307 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:12.082801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:12.082759 4828 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 17:18:12.083112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:12.083064 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" podUID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 17:18:12.107712 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:12.107640 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" podStartSLOduration=2.107618211 podStartE2EDuration="2.107618211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:12.094917342 +0000 UTC m=+1869.740427520" watchObservedRunningTime="2026-10-30 17:18:12.107618211 +0000 UTC m=+1869.753128392" Oct 30 17:18:12.547985 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:18:12.547535 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafe970bc_4f76_41e9_93c1_f18b36259ae3.slice/crio-6c227b91b00c5e12fe13f0049dbc11c5b18aa017d7737d3ee4c5df8bebf72d04 WatchSource:0}: Error finding container 6c227b91b00c5e12fe13f0049dbc11c5b18aa017d7737d3ee4c5df8bebf72d04: Status 404 returned error can't find the container with id 6c227b91b00c5e12fe13f0049dbc11c5b18aa017d7737d3ee4c5df8bebf72d04 Oct 30 17:18:12.559884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:12.559851 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22"] Oct 30 17:18:13.021445 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:13.021398 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" event={"ID":"afe970bc-4f76-41e9-93c1-f18b36259ae3","Type":"ContainerStarted","Data":"d9eebbeafd0c1c0fed3842d6b8f5e3e982994a096533997e091a4bff35f1d9a0"} Oct 30 17:18:13.021643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:13.021461 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" event={"ID":"afe970bc-4f76-41e9-93c1-f18b36259ae3","Type":"ContainerStarted","Data":"6c227b91b00c5e12fe13f0049dbc11c5b18aa017d7737d3ee4c5df8bebf72d04"} Oct 30 17:18:13.024136 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:13.024095 4828 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 17:18:13.024283 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:13.024158 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" podUID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 17:18:13.027237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:13.027215 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:13.067492 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:13.067440 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" podStartSLOduration=2.0674258070000002 podStartE2EDuration="2.067425807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:13.065120606 +0000 UTC m=+1870.710630786" watchObservedRunningTime="2026-10-30 17:18:13.067425807 +0000 UTC m=+1870.712935986" Oct 30 17:18:13.381796 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:13.381696 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:14.024830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:14.024427 4828 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 17:18:14.024830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:14.024491 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" podUID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 17:18:15.024476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:15.024440 4828 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 17:18:15.024659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:15.024498 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" podUID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 17:18:15.189924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:15.188793 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:16.024819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:16.024784 4828 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 17:18:16.025062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:16.024844 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" podUID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 17:18:17.024504 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:17.024468 4828 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 17:18:17.024994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:17.024522 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" podUID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 17:18:17.074927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:17.064287 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:17.922707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:17.922251 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km"] Oct 30 17:18:17.928360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:17.928333 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:17.950945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:17.950912 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km"] Oct 30 17:18:17.955242 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:17.955208 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e5ac8f\"/\"test-model-e5ac8f-kserve-self-signed-certs\"" Oct 30 17:18:17.955488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:17.955472 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e5ac8f\"/\"default-dockercfg-cm4dh\"" Oct 30 17:18:17.958173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:17.957881 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e5ac8f\"/\"openshift-service-ca.crt\"" Oct 30 17:18:17.958173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:17.958119 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e5ac8f\"/\"kube-root-ca.crt\"" Oct 30 17:18:18.013844 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.013785 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-dshm\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.014094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.013976 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-home\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.014236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.014121 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-model-cache\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.014236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.014164 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8f899f-225a-4a7a-a938-41c23e83291e-tls-certs\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.014383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.014233 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-tmp-dir\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.014383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.014260 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6rq9\" (UniqueName: \"kubernetes.io/projected/0d8f899f-225a-4a7a-a938-41c23e83291e-kube-api-access-m6rq9\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.024642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.024613 4828 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 17:18:18.025052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.024663 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" podUID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 17:18:18.027202 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.027174 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:18.115812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.115773 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-model-cache\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.116044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.115823 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8f899f-225a-4a7a-a938-41c23e83291e-tls-certs\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.116044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.115876 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-tmp-dir\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.116044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.115894 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6rq9\" (UniqueName: \"kubernetes.io/projected/0d8f899f-225a-4a7a-a938-41c23e83291e-kube-api-access-m6rq9\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.116263 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.116146 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-dshm\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.116263 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.116227 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-home\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.116488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.116379 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-model-cache\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.116565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.116507 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-tmp-dir\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.116629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.116611 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-home\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.118566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.118543 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-dshm\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.118985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.118963 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8f899f-225a-4a7a-a938-41c23e83291e-tls-certs\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.140622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.140577 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6rq9\" (UniqueName: \"kubernetes.io/projected/0d8f899f-225a-4a7a-a938-41c23e83291e-kube-api-access-m6rq9\") pod \"test-model-e5ac8f-kserve-69658fc9d7-826km\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.254982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.253829 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:18.901471 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:18.901420 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km"] Oct 30 17:18:18.918943 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:18:18.918913 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d8f899f_225a_4a7a_a938_41c23e83291e.slice/crio-760d090410ac12b283f11cf3e4b159b7e2911721a01802c035141e174995a888 WatchSource:0}: Error finding container 760d090410ac12b283f11cf3e4b159b7e2911721a01802c035141e174995a888: Status 404 returned error can't find the container with id 760d090410ac12b283f11cf3e4b159b7e2911721a01802c035141e174995a888 Oct 30 17:18:19.030299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:19.030265 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:19.030767 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:19.030710 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:19.031612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:19.031589 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:18:19.086955 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:19.086874 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" event={"ID":"0d8f899f-225a-4a7a-a938-41c23e83291e","Type":"ContainerStarted","Data":"90e6b393b527a3d50204e5756363162fdf43816a26dee50aedc6181c4e70f095"} Oct 30 17:18:19.087120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:19.086967 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" event={"ID":"0d8f899f-225a-4a7a-a938-41c23e83291e","Type":"ContainerStarted","Data":"760d090410ac12b283f11cf3e4b159b7e2911721a01802c035141e174995a888"} Oct 30 17:18:19.255071 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:19.254973 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:19.360430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:19.360392 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:21.193944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:21.191983 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:21.869512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:21.869471 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:22.033919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:22.028300 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:22.237506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:22.237012 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:22.259063 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:22.258947 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:18:22.281329 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:22.281261 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" podStartSLOduration=5.281243583 podStartE2EDuration="5.281243583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:19.185526818 +0000 UTC m=+1876.831037000" watchObservedRunningTime="2026-10-30 17:18:22.281243583 +0000 UTC m=+1879.926753761" Oct 30 17:18:22.814916 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:22.814347 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:23.246998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:23.246916 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:23.716401 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:23.716365 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:24.325188 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:24.325154 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:24.606827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:24.606733 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:24.779115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:24.776471 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:26.544874 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:26.544833 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:28.254324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:28.254288 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:28.268720 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:28.268688 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:28.374953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:28.374867 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:28.567561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:28.567521 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:29.222543 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:29.222509 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:18:30.127434 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:30.127390 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:30.287434 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:30.287397 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:30.329493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:30.329440 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb"] Oct 30 17:18:30.341795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:30.336467 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" Oct 30 17:18:30.529483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:30.455575 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c45877\"" Oct 30 17:18:30.529483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:30.456301 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c45877-dockercfg-v4h46\"" Oct 30 17:18:30.758450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:30.758409 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb\" (UID: \"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" Oct 30 17:18:30.758769 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:30.758748 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwjp\" (UniqueName: \"kubernetes.io/projected/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-kube-api-access-6dwjp\") pod \"payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb\" (UID: \"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" Oct 30 17:18:30.761678 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:30.761619 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb"] Oct 30 17:18:30.861141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:30.861057 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb\" (UID: \"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" Oct 30 17:18:30.861141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:30.861121 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwjp\" (UniqueName: \"kubernetes.io/projected/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-kube-api-access-6dwjp\") pod \"payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb\" (UID: \"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" Oct 30 17:18:30.861851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:30.861824 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb\" (UID: \"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" Oct 30 17:18:30.885747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:30.885705 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwjp\" (UniqueName: \"kubernetes.io/projected/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-kube-api-access-6dwjp\") pod \"payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb\" (UID: \"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" Oct 30 17:18:31.102606 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:31.102565 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:31.113167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:31.104215 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" Oct 30 17:18:31.468825 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:18:31.468788 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0c1cf4_fc08_47bb_b0a4_493f89042c1e.slice/crio-cb4f034d3bc48e15ec85c2fd717dfed9971fa0615ddd94b5be845db6e7983a29 WatchSource:0}: Error finding container cb4f034d3bc48e15ec85c2fd717dfed9971fa0615ddd94b5be845db6e7983a29: Status 404 returned error can't find the container with id cb4f034d3bc48e15ec85c2fd717dfed9971fa0615ddd94b5be845db6e7983a29 Oct 30 17:18:31.470932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:31.470888 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb"] Oct 30 17:18:32.250513 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:32.250476 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" event={"ID":"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e","Type":"ContainerStarted","Data":"49f9a1eb6acd01d121d64439d094f0a123857e1d20295c3c2af5546351febf68"} Oct 30 17:18:32.250692 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:32.250540 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" event={"ID":"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e","Type":"ContainerStarted","Data":"cb4f034d3bc48e15ec85c2fd717dfed9971fa0615ddd94b5be845db6e7983a29"} Oct 30 17:18:32.250692 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:32.250575 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" Oct 30 17:18:32.302270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:32.302177 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" podStartSLOduration=2.302163343 podStartE2EDuration="2.302163343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:32.296501757 +0000 UTC m=+1889.942011961" watchObservedRunningTime="2026-10-30 17:18:32.302163343 +0000 UTC m=+1889.947673523" Oct 30 17:18:32.876010 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:32.871184 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:34.473925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:34.468016 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:36.264781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:36.264739 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:37.904170 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:37.904080 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:37.987789 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:18:37.987725 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:18:39.510715 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:39.510677 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:41.287368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:41.287326 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:43.102028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:43.095388 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:43.262338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:43.262309 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" Oct 30 17:18:44.727922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:44.726088 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:44.918930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:44.917571 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:46.554474 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:46.554442 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:46.707605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:46.707567 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:47.132964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:47.132892 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:47.679674 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:47.678125 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:47.825658 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:47.822645 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:18:47.863039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:47.862081 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:49.260925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:49.259933 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" podUID="e14ba43d-a967-4948-9b5e-9258ed521f08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.174:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:18:49.447957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:49.447006 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:18:49.679987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:49.670730 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:18:49.814541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:49.814491 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:18:49.964431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:49.959619 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:51.372149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.372107 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:51.519729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.519698 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:51.613819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.613776 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs"] Oct 30 17:18:51.620152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.620125 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.638370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.638336 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs"] Oct 30 17:18:51.643458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.643430 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-82243b\"/\"auth-test-model-82243b-kserve-self-signed-certs\"" Oct 30 17:18:51.644103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.644081 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-82243b\"/\"openshift-service-ca.crt\"" Oct 30 17:18:51.644321 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.644288 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-model-cache\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.644424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.644325 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-82243b\"/\"kube-root-ca.crt\"" Oct 30 17:18:51.644492 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.644468 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkvjc\" (UniqueName: \"kubernetes.io/projected/92fad2ed-7234-450c-93cc-7c424235140d-kube-api-access-qkvjc\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.644555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.644499 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-home\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.644555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.644527 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-tmp-dir\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.644867 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.644844 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-dshm\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.644983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.644894 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fad2ed-7234-450c-93cc-7c424235140d-tls-certs\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.693599 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.693557 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:51.734007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.733977 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-82243b\"/\"default-dockercfg-x4577\"" Oct 30 17:18:51.746857 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.746825 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-dshm\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.747090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.746866 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fad2ed-7234-450c-93cc-7c424235140d-tls-certs\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.747090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.746962 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-model-cache\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.747090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.747067 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkvjc\" (UniqueName: \"kubernetes.io/projected/92fad2ed-7234-450c-93cc-7c424235140d-kube-api-access-qkvjc\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.747090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.747091 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-home\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.747334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.747111 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-tmp-dir\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.747446 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.747422 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-model-cache\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.747510 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.747466 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-home\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.747629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.747609 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-tmp-dir\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.749439 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.749416 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-dshm\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.749603 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.749585 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fad2ed-7234-450c-93cc-7c424235140d-tls-certs\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.771342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.771311 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkvjc\" (UniqueName: \"kubernetes.io/projected/92fad2ed-7234-450c-93cc-7c424235140d-kube-api-access-qkvjc\") pod \"auth-test-model-82243b-kserve-5cc4bdc969-672rs\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:51.943981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:51.941213 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:52.271754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:52.271717 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:52.447924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:52.441578 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:52.738766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:52.738735 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs"] Oct 30 17:18:52.819845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:52.819817 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" event={"ID":"92fad2ed-7234-450c-93cc-7c424235140d","Type":"ContainerStarted","Data":"87329eb93564d643b922d7b0b35fa8c58d814af110274123ab9dcb578a86a47a"} Oct 30 17:18:53.830523 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:53.830482 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" event={"ID":"92fad2ed-7234-450c-93cc-7c424235140d","Type":"ContainerStarted","Data":"8118473804a2ba3603e94b95a5d03b43fc8f3b482e22c19709b0f84585b3dbf7"} Oct 30 17:18:53.879463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:53.879402 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" podStartSLOduration=2.8793872499999997 podStartE2EDuration="2.87938725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:53.873209732 +0000 UTC m=+1911.518719918" watchObservedRunningTime="2026-10-30 17:18:53.87938725 +0000 UTC m=+1911.524897431" Oct 30 17:18:53.941702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:53.941672 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:18:54.243419 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:54.243334 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:54.527961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:54.500134 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:56.212684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:56.212645 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:57.519497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:57.519462 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:57.793763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:57.793666 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:57.982994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:57.982892 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:18:59.784927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:18:59.775779 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:01.469933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:01.467474 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:01.968873 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:01.968835 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:19:01.983764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:01.983731 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:19:02.955020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:02.954983 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:19:03.041538 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:03.041504 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:04.908215 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:04.908179 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:06.624486 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:06.624438 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:07.915781 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:19:07.915743 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:19:08.248973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:08.239840 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:09.237521 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:09.237481 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:09.390301 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:09.390263 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:10.098735 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:10.098691 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:11.884841 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:11.884797 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:13.668501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:13.668454 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:15.073134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:15.073035 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:15.232018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:15.231984 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:15.535359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:15.535299 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:17.098949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.098884 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc"] Oct 30 17:19:17.105542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.105516 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.124245 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.124214 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c45877-serving-cert\"" Oct 30 17:19:17.126305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.126274 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc"] Oct 30 17:19:17.171600 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.171561 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-metrics-certs\") pod \"maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.171939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.171913 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr8gp\" (UniqueName: \"kubernetes.io/projected/99acb5e4-3966-4cc9-a242-59c620b065f9-kube-api-access-fr8gp\") pod \"maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.172060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.172043 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.172129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.172079 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99acb5e4-3966-4cc9-a242-59c620b065f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.248541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.248490 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:17.273443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.273408 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr8gp\" (UniqueName: \"kubernetes.io/projected/99acb5e4-3966-4cc9-a242-59c620b065f9-kube-api-access-fr8gp\") pod \"maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.273623 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.273515 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.273623 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.273542 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99acb5e4-3966-4cc9-a242-59c620b065f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.273623 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.273591 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-metrics-certs\") pod \"maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.274350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.274323 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99acb5e4-3966-4cc9-a242-59c620b065f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.276251 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.276230 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.276406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.276376 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-metrics-certs\") pod \"maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.352371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.352282 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr8gp\" (UniqueName: \"kubernetes.io/projected/99acb5e4-3966-4cc9-a242-59c620b065f9-kube-api-access-fr8gp\") pod \"maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.425511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.425219 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:17.435821 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.435785 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:17.786946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:17.786912 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc"] Oct 30 17:19:17.787488 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:19:17.787461 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99acb5e4_3966_4cc9_a242_59c620b065f9.slice/crio-5588d20720e20d9d5be46c444c0d7ae631b225423edf9d2e8fc28dc37a05bcac WatchSource:0}: Error finding container 5588d20720e20d9d5be46c444c0d7ae631b225423edf9d2e8fc28dc37a05bcac: Status 404 returned error can't find the container with id 5588d20720e20d9d5be46c444c0d7ae631b225423edf9d2e8fc28dc37a05bcac Oct 30 17:19:18.195492 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:18.195389 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" event={"ID":"99acb5e4-3966-4cc9-a242-59c620b065f9","Type":"ContainerStarted","Data":"5588d20720e20d9d5be46c444c0d7ae631b225423edf9d2e8fc28dc37a05bcac"} Oct 30 17:19:19.106030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:19.105989 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:19.207997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:19.207936 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" event={"ID":"99acb5e4-3966-4cc9-a242-59c620b065f9","Type":"ContainerStarted","Data":"71b1a640f0f2818079bde52e4ef9c8dba7c7f27b67a07076799d119551d43709"} Oct 30 17:19:19.208391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:19.208013 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:19.254770 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:19.254715 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" podStartSLOduration=1.7599463659999999 podStartE2EDuration="2.254701492s" totalImagesPullingTime="494.755126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:19.250970082 +0000 UTC m=+1936.896480263" watchObservedRunningTime="2026-10-30 17:19:19.254701492 +0000 UTC m=+1936.900211672" Oct 30 17:19:20.938483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:20.938443 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:22.675921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:22.673701 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:24.358532 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:24.358434 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:25.224506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:25.224475 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:19:26.165946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:26.165877 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:27.880657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:27.878217 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:29.494807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:29.494773 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:31.220481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:31.220441 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:32.897924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:32.871853 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:34.437437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:34.437389 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:36.249291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:36.249030 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:37.918926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:37.916050 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:38.003933 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:19:38.001741 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:19:39.496683 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.496645 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5"] Oct 30 17:19:39.501683 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.501659 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:39.525793 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.525756 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-001a96-serving-cert\"" Oct 30 17:19:39.527046 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.527021 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5"] Oct 30 17:19:39.561985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.561951 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:39.562258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.562233 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjzcj\" (UniqueName: \"kubernetes.io/projected/694fc7b7-12ff-4f40-a513-44fd85f92439-kube-api-access-mjzcj\") pod \"maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:39.562321 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.562286 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:39.562412 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.562396 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/694fc7b7-12ff-4f40-a513-44fd85f92439-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:39.623531 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.623490 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:39.663877 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.663836 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjzcj\" (UniqueName: \"kubernetes.io/projected/694fc7b7-12ff-4f40-a513-44fd85f92439-kube-api-access-mjzcj\") pod \"maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:39.664095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.663888 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:39.664095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.663966 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/694fc7b7-12ff-4f40-a513-44fd85f92439-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:39.664197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.664131 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:39.664800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.664778 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/694fc7b7-12ff-4f40-a513-44fd85f92439-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:39.666705 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.666681 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:39.666834 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.666745 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:39.688847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.688814 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjzcj\" (UniqueName: \"kubernetes.io/projected/694fc7b7-12ff-4f40-a513-44fd85f92439-kube-api-access-mjzcj\") pod \"maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:39.804928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.800242 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:39.830924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:39.824216 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:40.182168 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:19:40.182136 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694fc7b7_12ff_4f40_a513_44fd85f92439.slice/crio-3fa6e41a294fb89a794a6b481feb2e593e0522665722b9584cd42e8e16c1073f WatchSource:0}: Error finding container 3fa6e41a294fb89a794a6b481feb2e593e0522665722b9584cd42e8e16c1073f: Status 404 returned error can't find the container with id 3fa6e41a294fb89a794a6b481feb2e593e0522665722b9584cd42e8e16c1073f Oct 30 17:19:40.184584 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:40.184535 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5"] Oct 30 17:19:40.490024 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:40.486058 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" event={"ID":"694fc7b7-12ff-4f40-a513-44fd85f92439","Type":"ContainerStarted","Data":"3fa6e41a294fb89a794a6b481feb2e593e0522665722b9584cd42e8e16c1073f"} Oct 30 17:19:41.574001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:41.561591 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" event={"ID":"694fc7b7-12ff-4f40-a513-44fd85f92439","Type":"ContainerStarted","Data":"b768bbb928cd64c9c04771d934a87bc520c6c8a19044ae78ee71e759ef0821f5"} Oct 30 17:19:41.574001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:41.562986 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:41.712268 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:41.712200 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" podStartSLOduration=2.247339385 podStartE2EDuration="2.712176974s" totalImagesPullingTime="464.837589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:41.675667889 +0000 UTC m=+1959.321178071" watchObservedRunningTime="2026-10-30 17:19:41.712176974 +0000 UTC m=+1959.357687155" Oct 30 17:19:42.240045 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:42.239993 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" podUID="afe970bc-4f76-41e9-93c1-f18b36259ae3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:42.250009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:42.249967 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" podUID="e14ba43d-a967-4948-9b5e-9258ed521f08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.174:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:42.298507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:42.297383 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:42.741926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:42.733614 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:44.349927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:44.343667 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:46.031957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:46.029966 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:47.771794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:47.771755 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:48.587067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:48.587041 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:19:49.483921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:49.482731 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:51.085925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:51.082159 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:52.775982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:52.775941 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:54.504539 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:54.504501 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:56.376924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:56.369526 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:57.655044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:57.655000 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:58.068648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:58.066891 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:19:59.628968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:19:59.628926 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:01.494616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:01.494558 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:03.068165 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:03.068130 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:04.695978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:04.693310 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:06.380924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:06.376081 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:08.216134 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:20:08.216089 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:20:08.234075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:08.234036 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:10.036225 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:10.036171 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:10.541747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:10.541710 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5"] Oct 30 17:20:10.542015 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:10.541988 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" podUID="ce315878-d81e-42b8-89d5-26086c747168" containerName="istio-proxy" containerID="cri-o://2feac357f730725672e7c5f6aa443ae8b6836c350c7031592120df5b6014af02" gracePeriod=30 Oct 30 17:20:10.611009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:10.610962 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:10.796215 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:10.796124 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2"] Oct 30 17:20:10.796436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:10.796410 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" containerID="cri-o://91836df686bcd8c82b6e413ecd67633604b87ad46e44334a8730d8c23078d275" gracePeriod=30 Oct 30 17:20:11.528797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:11.528762 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2"] Oct 30 17:20:11.529376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:11.528993 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" containerID="cri-o://6d04aa9a1a97dacd004f2d54dde6a4a54e3e66900768a95a23962043beb493bb" gracePeriod=30 Oct 30 17:20:11.744267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:11.744232 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:12.596381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.596340 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb"] Oct 30 17:20:12.603300 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.603271 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.633528 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.633485 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb"] Oct 30 17:20:12.637513 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.637485 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-42d381\"/\"openshift-service-ca.crt\"" Oct 30 17:20:12.637774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.637756 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-42d381\"/\"kube-root-ca.crt\"" Oct 30 17:20:12.637921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.637620 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-42d381\"/\"default-dockercfg-tp7f4\"" Oct 30 17:20:12.637921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.637539 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-42d381\"/\"test-model-42d381-kserve-self-signed-certs\"" Oct 30 17:20:12.690360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.690318 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkbpf\" (UniqueName: \"kubernetes.io/projected/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-kube-api-access-hkbpf\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.690560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.690451 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-model-cache\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.690681 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.690635 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tmp-dir\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.690765 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.690684 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-home\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.690972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.690946 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-dshm\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.691059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.690985 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tls-certs\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.695027 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.694993 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:12.792293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.792246 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkbpf\" (UniqueName: \"kubernetes.io/projected/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-kube-api-access-hkbpf\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.792511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.792346 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-model-cache\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.792511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.792457 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tmp-dir\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.792511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.792485 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-home\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.792679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.792647 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-dshm\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.792740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.792673 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tls-certs\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.792864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.792837 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-model-cache\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.793079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.792892 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tmp-dir\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.793157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.792986 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-home\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.795084 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.795053 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-dshm\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.795411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.795394 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tls-certs\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.818646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.818616 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkbpf\" (UniqueName: \"kubernetes.io/projected/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-kube-api-access-hkbpf\") pod \"test-model-42d381-kserve-5d5bbb5f79-nl6kb\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:12.949056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:12.948955 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:13.368819 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:20:13.368785 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1a87552_a6e2_451c_bd7c_e8b781c48aa4.slice/crio-d0a141bdb69f5c81d228815f3046f9504914c0b2878f82a114eed6a494aca7d1 WatchSource:0}: Error finding container d0a141bdb69f5c81d228815f3046f9504914c0b2878f82a114eed6a494aca7d1: Status 404 returned error can't find the container with id d0a141bdb69f5c81d228815f3046f9504914c0b2878f82a114eed6a494aca7d1 Oct 30 17:20:13.372104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:13.372078 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb"] Oct 30 17:20:13.672985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:13.672247 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:13.954127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:13.954040 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" event={"ID":"b1a87552-a6e2-451c-bd7c-e8b781c48aa4","Type":"ContainerStarted","Data":"fcdfe6b5498ee744b0e80b4bba88bb681f2b694e1dc93b8411fd1a7efd6ef9aa"} Oct 30 17:20:13.954127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:13.954081 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" event={"ID":"b1a87552-a6e2-451c-bd7c-e8b781c48aa4","Type":"ContainerStarted","Data":"d0a141bdb69f5c81d228815f3046f9504914c0b2878f82a114eed6a494aca7d1"} Oct 30 17:20:13.999724 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:13.999671 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" podStartSLOduration=1.999656839 podStartE2EDuration="1.999656839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:13.999337117 +0000 UTC m=+1991.644847310" watchObservedRunningTime="2026-10-30 17:20:13.999656839 +0000 UTC m=+1991.645167018" Oct 30 17:20:14.949938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:14.949478 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:15.299378 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:15.276477 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:15.939658 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:15.939622 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:20:15.977420 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:15.977385 4828 generic.go:356] "Generic (PLEG): container finished" podID="ce315878-d81e-42b8-89d5-26086c747168" containerID="2feac357f730725672e7c5f6aa443ae8b6836c350c7031592120df5b6014af02" exitCode=0 Oct 30 17:20:15.977939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:15.977450 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" Oct 30 17:20:15.977939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:15.977484 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" event={"ID":"ce315878-d81e-42b8-89d5-26086c747168","Type":"ContainerDied","Data":"2feac357f730725672e7c5f6aa443ae8b6836c350c7031592120df5b6014af02"} Oct 30 17:20:15.978080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:15.978025 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5" event={"ID":"ce315878-d81e-42b8-89d5-26086c747168","Type":"ContainerDied","Data":"07075faa715ad2d960d9658385b66b5db895f0e3af3c20e68890cc58521c88d5"} Oct 30 17:20:15.978140 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:15.978078 4828 scope.go:122] "RemoveContainer" containerID="2feac357f730725672e7c5f6aa443ae8b6836c350c7031592120df5b6014af02" Oct 30 17:20:15.980076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:15.980054 4828 generic.go:356] "Generic (PLEG): container finished" podID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerID="91836df686bcd8c82b6e413ecd67633604b87ad46e44334a8730d8c23078d275" exitCode=0 Oct 30 17:20:15.980246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:15.980220 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" event={"ID":"f5d62d2b-8034-4de7-a7f4-a8e8f6974679","Type":"ContainerDied","Data":"91836df686bcd8c82b6e413ecd67633604b87ad46e44334a8730d8c23078d275"} Oct 30 17:20:16.002025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.001236 4828 scope.go:122] "RemoveContainer" containerID="2feac357f730725672e7c5f6aa443ae8b6836c350c7031592120df5b6014af02" Oct 30 17:20:16.002025 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:20:16.001667 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2feac357f730725672e7c5f6aa443ae8b6836c350c7031592120df5b6014af02\": container with ID starting with 2feac357f730725672e7c5f6aa443ae8b6836c350c7031592120df5b6014af02 not found: ID does not exist" containerID="2feac357f730725672e7c5f6aa443ae8b6836c350c7031592120df5b6014af02" Oct 30 17:20:16.002025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.001704 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2feac357f730725672e7c5f6aa443ae8b6836c350c7031592120df5b6014af02"} err="failed to get container status \"2feac357f730725672e7c5f6aa443ae8b6836c350c7031592120df5b6014af02\": rpc error: code = NotFound desc = could not find container \"2feac357f730725672e7c5f6aa443ae8b6836c350c7031592120df5b6014af02\": container with ID starting with 2feac357f730725672e7c5f6aa443ae8b6836c350c7031592120df5b6014af02 not found: ID does not exist" Oct 30 17:20:16.075923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.075800 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-kube-api-access-dfdkq\" (UniqueName: \"kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-kube-api-access-dfdkq\") pod \"ce315878-d81e-42b8-89d5-26086c747168\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " Oct 30 17:20:16.076146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.076082 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-credential-socket\") pod \"ce315878-d81e-42b8-89d5-26086c747168\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " Oct 30 17:20:16.076146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.076130 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce315878-d81e-42b8-89d5-26086c747168-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce315878-d81e-42b8-89d5-26086c747168-istiod-ca-cert\") pod \"ce315878-d81e-42b8-89d5-26086c747168\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " Oct 30 17:20:16.076391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.076367 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-data\") pod \"ce315878-d81e-42b8-89d5-26086c747168\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " Oct 30 17:20:16.076498 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.076436 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce315878-d81e-42b8-89d5-26086c747168-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce315878-d81e-42b8-89d5-26086c747168-istio-podinfo\") pod \"ce315878-d81e-42b8-89d5-26086c747168\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " Oct 30 17:20:16.077326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.076563 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-socket\") pod \"ce315878-d81e-42b8-89d5-26086c747168\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " Oct 30 17:20:16.077326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.076618 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-certs\") pod \"ce315878-d81e-42b8-89d5-26086c747168\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " Oct 30 17:20:16.077326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.076658 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-istio-token\" (UniqueName: \"kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-istio-token\") pod \"ce315878-d81e-42b8-89d5-26086c747168\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " Oct 30 17:20:16.077326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.076718 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-envoy\") pod \"ce315878-d81e-42b8-89d5-26086c747168\" (UID: \"ce315878-d81e-42b8-89d5-26086c747168\") " Oct 30 17:20:16.078581 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.078545 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-data" pod "ce315878-d81e-42b8-89d5-26086c747168" (UID: "ce315878-d81e-42b8-89d5-26086c747168"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.079223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.079194 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-certs" pod "ce315878-d81e-42b8-89d5-26086c747168" (UID: "ce315878-d81e-42b8-89d5-26086c747168"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.080052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.080023 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-socket" pod "ce315878-d81e-42b8-89d5-26086c747168" (UID: "ce315878-d81e-42b8-89d5-26086c747168"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.080383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.080359 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce315878-d81e-42b8-89d5-26086c747168-istiod-ca-cert" pod "ce315878-d81e-42b8-89d5-26086c747168" (UID: "ce315878-d81e-42b8-89d5-26086c747168"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:20:16.080598 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.080575 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-credential-socket" pod "ce315878-d81e-42b8-89d5-26086c747168" (UID: "ce315878-d81e-42b8-89d5-26086c747168"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.082637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.082609 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-kube-api-access-dfdkq" pod "ce315878-d81e-42b8-89d5-26086c747168" (UID: "ce315878-d81e-42b8-89d5-26086c747168"). InnerVolumeSpecName "kube-api-access-dfdkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:16.083422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.083390 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce315878-d81e-42b8-89d5-26086c747168-istio-podinfo" pod "ce315878-d81e-42b8-89d5-26086c747168" (UID: "ce315878-d81e-42b8-89d5-26086c747168"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:20:16.083989 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.083949 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-istio-token" pod "ce315878-d81e-42b8-89d5-26086c747168" (UID: "ce315878-d81e-42b8-89d5-26086c747168"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:16.085079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.085053 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-envoy" pod "ce315878-d81e-42b8-89d5-26086c747168" (UID: "ce315878-d81e-42b8-89d5-26086c747168"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.141701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.141673 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:20:16.179146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.179104 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfdkq\" (UniqueName: \"kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-kube-api-access-dfdkq\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.179146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.179138 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.179146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.179149 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce315878-d81e-42b8-89d5-26086c747168-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.179146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.179158 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.179430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.179167 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce315878-d81e-42b8-89d5-26086c747168-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.179430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.179175 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.179430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.179183 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.179430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.179193 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce315878-d81e-42b8-89d5-26086c747168-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.179430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.179201 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce315878-d81e-42b8-89d5-26086c747168-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.279692 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.279652 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-socket\") pod \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " Oct 30 17:20:16.279942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.279705 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-envoy\") pod \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " Oct 30 17:20:16.279942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.279866 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-certs\") pod \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " Oct 30 17:20:16.280079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.280020 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-token\" (UniqueName: \"kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-token\") pod \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " Oct 30 17:20:16.280079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.280031 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-socket" pod "f5d62d2b-8034-4de7-a7f4-a8e8f6974679" (UID: "f5d62d2b-8034-4de7-a7f4-a8e8f6974679"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.280190 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.280104 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-kube-api-access-npphz\" (UniqueName: \"kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-kube-api-access-npphz\") pod \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " Oct 30 17:20:16.280243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.280205 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istiod-ca-cert\") pod \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " Oct 30 17:20:16.280294 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.280266 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-data\") pod \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " Oct 30 17:20:16.280348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.280328 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-podinfo\") pod \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " Oct 30 17:20:16.281687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.280435 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-credential-socket\") pod \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\" (UID: \"f5d62d2b-8034-4de7-a7f4-a8e8f6974679\") " Oct 30 17:20:16.281687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.281239 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.281687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.281294 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-certs" pod "f5d62d2b-8034-4de7-a7f4-a8e8f6974679" (UID: "f5d62d2b-8034-4de7-a7f4-a8e8f6974679"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.281687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.281371 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-data" pod "f5d62d2b-8034-4de7-a7f4-a8e8f6974679" (UID: "f5d62d2b-8034-4de7-a7f4-a8e8f6974679"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.281687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.281451 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-credential-socket" pod "f5d62d2b-8034-4de7-a7f4-a8e8f6974679" (UID: "f5d62d2b-8034-4de7-a7f4-a8e8f6974679"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.281687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.281638 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istiod-ca-cert" pod "f5d62d2b-8034-4de7-a7f4-a8e8f6974679" (UID: "f5d62d2b-8034-4de7-a7f4-a8e8f6974679"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:20:16.283340 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.283304 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-kube-api-access-npphz" pod "f5d62d2b-8034-4de7-a7f4-a8e8f6974679" (UID: "f5d62d2b-8034-4de7-a7f4-a8e8f6974679"). InnerVolumeSpecName "kube-api-access-npphz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:16.283461 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.283439 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-envoy" pod "f5d62d2b-8034-4de7-a7f4-a8e8f6974679" (UID: "f5d62d2b-8034-4de7-a7f4-a8e8f6974679"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.287369 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.287314 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-token" pod "f5d62d2b-8034-4de7-a7f4-a8e8f6974679" (UID: "f5d62d2b-8034-4de7-a7f4-a8e8f6974679"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:16.289016 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.288992 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-podinfo" pod "f5d62d2b-8034-4de7-a7f4-a8e8f6974679" (UID: "f5d62d2b-8034-4de7-a7f4-a8e8f6974679"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:20:16.364622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.364533 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5"] Oct 30 17:20:16.383472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.383433 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.383472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.383474 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.383721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.383487 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.383721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.383504 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.383721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.383520 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.383721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.383536 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.383721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.383548 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.383721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.383562 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npphz\" (UniqueName: \"kubernetes.io/projected/f5d62d2b-8034-4de7-a7f4-a8e8f6974679-kube-api-access-npphz\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.384388 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.384357 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-h9wl5"] Oct 30 17:20:16.500021 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.499985 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce315878-d81e-42b8-89d5-26086c747168" path="/var/lib/kubelet/pods/ce315878-d81e-42b8-89d5-26086c747168/volumes" Oct 30 17:20:16.860296 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.860263 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:20:16.976452 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.976367 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:16.993492 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.993456 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-envoy\") pod \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " Oct 30 17:20:16.993972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.993550 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-podinfo\") pod \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " Oct 30 17:20:16.994071 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.994010 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-data\") pod \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " Oct 30 17:20:16.994131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.994089 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-socket\") pod \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " Oct 30 17:20:16.994131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.994117 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-kube-api-access-chzms\" (UniqueName: \"kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-kube-api-access-chzms\") pod \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " Oct 30 17:20:16.994295 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.994196 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-credential-socket\") pod \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " Oct 30 17:20:16.994414 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.994376 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-socket" pod "703615b6-f313-4e6d-a35c-6d9caf3d3ae9" (UID: "703615b6-f313-4e6d-a35c-6d9caf3d3ae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.994502 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.994436 4828 generic.go:356] "Generic (PLEG): container finished" podID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerID="6d04aa9a1a97dacd004f2d54dde6a4a54e3e66900768a95a23962043beb493bb" exitCode=0 Oct 30 17:20:16.994502 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.994494 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istiod-ca-cert\") pod \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " Oct 30 17:20:16.994653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.994579 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-certs\") pod \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " Oct 30 17:20:16.994653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.994599 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" Oct 30 17:20:16.994653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.994631 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-token\" (UniqueName: \"kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-token\") pod \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\" (UID: \"703615b6-f313-4e6d-a35c-6d9caf3d3ae9\") " Oct 30 17:20:16.994886 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.994855 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-data" pod "703615b6-f313-4e6d-a35c-6d9caf3d3ae9" (UID: "703615b6-f313-4e6d-a35c-6d9caf3d3ae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.995041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.995012 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" event={"ID":"703615b6-f313-4e6d-a35c-6d9caf3d3ae9","Type":"ContainerDied","Data":"6d04aa9a1a97dacd004f2d54dde6a4a54e3e66900768a95a23962043beb493bb"} Oct 30 17:20:16.995152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.995074 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2" event={"ID":"703615b6-f313-4e6d-a35c-6d9caf3d3ae9","Type":"ContainerDied","Data":"07a9426b3c5be9ea213c7da7b1d1be0a5a7d5fd3d5d362923ae7548e593e6f4a"} Oct 30 17:20:16.995152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.995106 4828 scope.go:122] "RemoveContainer" containerID="6d04aa9a1a97dacd004f2d54dde6a4a54e3e66900768a95a23962043beb493bb" Oct 30 17:20:16.995152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.995138 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-certs" pod "703615b6-f313-4e6d-a35c-6d9caf3d3ae9" (UID: "703615b6-f313-4e6d-a35c-6d9caf3d3ae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.995324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.995306 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-credential-socket" pod "703615b6-f313-4e6d-a35c-6d9caf3d3ae9" (UID: "703615b6-f313-4e6d-a35c-6d9caf3d3ae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:16.995516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.995495 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istiod-ca-cert" pod "703615b6-f313-4e6d-a35c-6d9caf3d3ae9" (UID: "703615b6-f313-4e6d-a35c-6d9caf3d3ae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:20:16.995945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.995872 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.995945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.995921 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.995945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.995939 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.996223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.995953 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.996223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.995967 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.997889 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.997785 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" Oct 30 17:20:16.997889 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:16.997823 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2" event={"ID":"f5d62d2b-8034-4de7-a7f4-a8e8f6974679","Type":"ContainerDied","Data":"0715873e285e4e7ab919fdc5dfe5d235dd4555009e5000a813ac87c2a8e78ffc"} Oct 30 17:20:17.003552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.003523 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-envoy" pod "703615b6-f313-4e6d-a35c-6d9caf3d3ae9" (UID: "703615b6-f313-4e6d-a35c-6d9caf3d3ae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:17.003664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.003586 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-kube-api-access-chzms" pod "703615b6-f313-4e6d-a35c-6d9caf3d3ae9" (UID: "703615b6-f313-4e6d-a35c-6d9caf3d3ae9"). InnerVolumeSpecName "kube-api-access-chzms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:17.003664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.003633 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-podinfo" pod "703615b6-f313-4e6d-a35c-6d9caf3d3ae9" (UID: "703615b6-f313-4e6d-a35c-6d9caf3d3ae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:20:17.004960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.004938 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-token" pod "703615b6-f313-4e6d-a35c-6d9caf3d3ae9" (UID: "703615b6-f313-4e6d-a35c-6d9caf3d3ae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:17.129749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.129147 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:17.129749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.129184 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chzms\" (UniqueName: \"kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-kube-api-access-chzms\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:17.129749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.129204 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:17.129749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.129221 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703615b6-f313-4e6d-a35c-6d9caf3d3ae9-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:17.133162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.133128 4828 scope.go:122] "RemoveContainer" containerID="6d04aa9a1a97dacd004f2d54dde6a4a54e3e66900768a95a23962043beb493bb" Oct 30 17:20:17.133628 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:20:17.133564 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d04aa9a1a97dacd004f2d54dde6a4a54e3e66900768a95a23962043beb493bb\": container with ID starting with 6d04aa9a1a97dacd004f2d54dde6a4a54e3e66900768a95a23962043beb493bb not found: ID does not exist" containerID="6d04aa9a1a97dacd004f2d54dde6a4a54e3e66900768a95a23962043beb493bb" Oct 30 17:20:17.133628 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.133607 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d04aa9a1a97dacd004f2d54dde6a4a54e3e66900768a95a23962043beb493bb"} err="failed to get container status \"6d04aa9a1a97dacd004f2d54dde6a4a54e3e66900768a95a23962043beb493bb\": rpc error: code = NotFound desc = could not find container \"6d04aa9a1a97dacd004f2d54dde6a4a54e3e66900768a95a23962043beb493bb\": container with ID starting with 6d04aa9a1a97dacd004f2d54dde6a4a54e3e66900768a95a23962043beb493bb not found: ID does not exist" Oct 30 17:20:17.133793 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.133634 4828 scope.go:122] "RemoveContainer" containerID="91836df686bcd8c82b6e413ecd67633604b87ad46e44334a8730d8c23078d275" Oct 30 17:20:17.135934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.135835 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2"] Oct 30 17:20:17.141494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.141467 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-nqsp2"] Oct 30 17:20:17.347202 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.347171 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2"] Oct 30 17:20:17.352456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:17.352427 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-sjbz2"] Oct 30 17:20:18.498028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:18.497993 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" path="/var/lib/kubelet/pods/703615b6-f313-4e6d-a35c-6d9caf3d3ae9/volumes" Oct 30 17:20:18.498596 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:18.498581 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" path="/var/lib/kubelet/pods/f5d62d2b-8034-4de7-a7f4-a8e8f6974679/volumes" Oct 30 17:20:18.846684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:18.846567 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:20.541273 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:20.541207 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:22.190209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:22.190170 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:22.949916 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:22.949858 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:22.963304 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:22.963271 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:23.066407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:23.066377 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:24.091700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:24.088287 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:25.623381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:25.623341 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:27.345398 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:27.345358 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:29.201934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:29.192582 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:31.002926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:31.002851 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:32.731594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:32.731557 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:34.357765 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:34.357681 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:34.586974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:34.580540 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:36.298922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:36.297803 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:37.766220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:37.766182 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:37.916377 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:20:37.916338 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:20:39.523689 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:39.523651 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:41.375845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:41.375805 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:41.721234 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:41.721121 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:42.160012 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:42.159970 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5"] Oct 30 17:20:42.235479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:42.235447 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:43.313377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:43.313328 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:20:43.662191 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:43.662093 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:43.799272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:43.799232 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:43.852837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:43.852793 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:44.052070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.052008 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" podUID="92fad2ed-7234-450c-93cc-7c424235140d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.179:8000/ready\": context deadline exceeded" Oct 30 17:20:44.067032 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.066985 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" podUID="b1a87552-a6e2-451c-bd7c-e8b781c48aa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.182:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:44.087148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.087101 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" podUID="b1a87552-a6e2-451c-bd7c-e8b781c48aa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:44.215272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.215167 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" podUID="0d8f899f-225a-4a7a-a938-41c23e83291e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.177:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:44.264926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.256016 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" podUID="afe970bc-4f76-41e9-93c1-f18b36259ae3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:44.279438 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.279008 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" podUID="e14ba43d-a967-4948-9b5e-9258ed521f08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.174:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:44.324270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.324112 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:44.325390 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.325301 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:20:44.467927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.455107 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:44.715068 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.676055 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:44.753013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.750359 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:44.825925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.800563 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:44.885926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.861459 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:20:44.909931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:44.906073 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:20:45.086926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:45.064150 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" podUID="92fad2ed-7234-450c-93cc-7c424235140d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.179:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:45.111999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:45.111964 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:20:45.117921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:45.088030 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" podUID="b1a87552-a6e2-451c-bd7c-e8b781c48aa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:45.118567 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:45.118543 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:45.224218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:45.224179 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" podUID="0d8f899f-225a-4a7a-a938-41c23e83291e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.177:8000/ready\": context deadline exceeded" Oct 30 17:20:45.230944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:45.226265 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:20:45.331265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:45.329224 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:45.441443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:45.441360 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:20:45.444267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:45.444239 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:20:45.445598 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:45.445508 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:20:45.448016 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:45.447994 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:20:45.455013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:45.454990 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:20:45.526314 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:45.526275 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:20:46.685660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:46.685567 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:46.866811 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:46.866777 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:48.472893 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:48.467107 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:50.074928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.069609 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:50.903922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.902786 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks"] Oct 30 17:20:50.904138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.904111 4828 cpu_manager.go:414] "containerMap: removing container" podUID="ce315878-d81e-42b8-89d5-26086c747168" containerName="istio-proxy" Oct 30 17:20:50.904138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.904131 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce315878-d81e-42b8-89d5-26086c747168" containerName="istio-proxy" Oct 30 17:20:50.904284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.904166 4828 cpu_manager.go:414] "containerMap: removing container" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" Oct 30 17:20:50.904284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.904174 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" Oct 30 17:20:50.904284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.904192 4828 cpu_manager.go:414] "containerMap: removing container" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" Oct 30 17:20:50.904284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.904199 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" Oct 30 17:20:50.904499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.904365 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce315878-d81e-42b8-89d5-26086c747168" containerName="istio-proxy" Oct 30 17:20:50.904499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.904381 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d62d2b-8034-4de7-a7f4-a8e8f6974679" containerName="istio-proxy" Oct 30 17:20:50.904499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.904391 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="703615b6-f313-4e6d-a35c-6d9caf3d3ae9" containerName="istio-proxy" Oct 30 17:20:50.910459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.910433 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:50.946174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.946127 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-987dfc\"/\"openshift-service-ca.crt\"" Oct 30 17:20:50.946359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.946273 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-987dfc\"/\"auth-test-model-987dfc-kserve-self-signed-certs\"" Oct 30 17:20:50.946359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.946351 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-987dfc\"/\"default-dockercfg-zc5jg\"" Oct 30 17:20:50.946542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.946521 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-987dfc\"/\"kube-root-ca.crt\"" Oct 30 17:20:50.955772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:50.955740 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks"] Oct 30 17:20:51.073721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.073683 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e47918-6c04-453d-b6ed-fabc181b3807-tls-certs\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.073951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.073820 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-tmp-dir\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.074013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.073992 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mvhs\" (UniqueName: \"kubernetes.io/projected/a5e47918-6c04-453d-b6ed-fabc181b3807-kube-api-access-5mvhs\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.074134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.074107 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-home\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.074258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.074230 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-model-cache\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.074316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.074263 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-dshm\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.102217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.102182 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:51.175655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.175557 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-home\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.175835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.175678 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-model-cache\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.175835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.175724 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-dshm\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.175980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.175920 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e47918-6c04-453d-b6ed-fabc181b3807-tls-certs\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.176057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.176021 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-tmp-dir\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.176115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.176042 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-home\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.176115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.176084 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-model-cache\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.176224 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.176113 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvhs\" (UniqueName: \"kubernetes.io/projected/a5e47918-6c04-453d-b6ed-fabc181b3807-kube-api-access-5mvhs\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.176375 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.176350 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-tmp-dir\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.178265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.178221 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-dshm\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.178708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.178685 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e47918-6c04-453d-b6ed-fabc181b3807-tls-certs\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.201385 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.201351 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mvhs\" (UniqueName: \"kubernetes.io/projected/a5e47918-6c04-453d-b6ed-fabc181b3807-kube-api-access-5mvhs\") pod \"auth-test-model-987dfc-kserve-5479d79d85-sh2ks\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.264921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.263685 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:51.803911 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:20:51.803855 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5e47918_6c04_453d_b6ed_fabc181b3807.slice/crio-51d7a389f207bee00c8642fb254b20451e54cdecc9553feea8f2e8a398290707 WatchSource:0}: Error finding container 51d7a389f207bee00c8642fb254b20451e54cdecc9553feea8f2e8a398290707: Status 404 returned error can't find the container with id 51d7a389f207bee00c8642fb254b20451e54cdecc9553feea8f2e8a398290707 Oct 30 17:20:51.808029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.807999 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks"] Oct 30 17:20:51.847992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:51.847950 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" event={"ID":"a5e47918-6c04-453d-b6ed-fabc181b3807","Type":"ContainerStarted","Data":"51d7a389f207bee00c8642fb254b20451e54cdecc9553feea8f2e8a398290707"} Oct 30 17:20:52.200179 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.198848 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:52.646312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.646270 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm"] Oct 30 17:20:52.652471 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.652443 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.668682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.668649 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-42d381-openshift-default-dockercfg-gf7bd\"" Oct 30 17:20:52.715781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.715714 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm"] Oct 30 17:20:52.759626 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.759590 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-data\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.759851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.759627 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-token\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.759851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.759705 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-credential-socket\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.760002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.759867 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce173a53-c21d-4b94-a7ec-d688705efe9e-istiod-ca-cert\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.760002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.759964 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-socket\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.760144 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.760054 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-podinfo\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.760257 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.760242 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpzsl\" (UniqueName: \"kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-kube-api-access-bpzsl\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.760301 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.760268 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-certs\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.760301 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.760298 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-envoy\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.858535 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.858498 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" event={"ID":"a5e47918-6c04-453d-b6ed-fabc181b3807","Type":"ContainerStarted","Data":"293a667a36317a929a86227ff25a002ee54562421317d060244df9c9bfee226f"} Oct 30 17:20:52.861667 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.861642 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpzsl\" (UniqueName: \"kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-kube-api-access-bpzsl\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.861807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.861676 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-certs\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.861807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.861705 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-envoy\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.861934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.861809 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-data\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.861934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.861837 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-token\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.861934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.861915 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-credential-socket\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.862115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.862043 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce173a53-c21d-4b94-a7ec-d688705efe9e-istiod-ca-cert\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.862115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.862084 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-socket\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.862241 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.862168 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-podinfo\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.862241 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.862187 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-certs\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.862364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.862283 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-data\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.862364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.862340 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-credential-socket\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.862513 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.862488 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-socket\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.862813 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.862793 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce173a53-c21d-4b94-a7ec-d688705efe9e-istiod-ca-cert\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.864304 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.864280 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-envoy\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.864644 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.864623 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-podinfo\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.886375 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.886333 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-token\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.886549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.886523 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpzsl\" (UniqueName: \"kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-kube-api-access-bpzsl\") pod \"e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:52.904941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.904811 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" podStartSLOduration=2.904797898 podStartE2EDuration="2.904797898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:52.901264485 +0000 UTC m=+2030.546774667" watchObservedRunningTime="2026-10-30 17:20:52.904797898 +0000 UTC m=+2030.550308124" Oct 30 17:20:52.969669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:52.969625 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:53.263990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:53.263958 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:20:53.395055 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:20:53.395021 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce173a53_c21d_4b94_a7ec_d688705efe9e.slice/crio-b171c61dd9adaf3c3ad622b98b186677f64c0a3950e51069a854b7e478ac0361 WatchSource:0}: Error finding container b171c61dd9adaf3c3ad622b98b186677f64c0a3950e51069a854b7e478ac0361: Status 404 returned error can't find the container with id b171c61dd9adaf3c3ad622b98b186677f64c0a3950e51069a854b7e478ac0361 Oct 30 17:20:53.395212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:53.395191 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm"] Oct 30 17:20:53.399005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:53.398966 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:20:53.399103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:53.399053 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:20:53.399103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:53.399096 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:20:53.869569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:53.853235 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:53.925022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:53.924985 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" event={"ID":"ce173a53-c21d-4b94-a7ec-d688705efe9e","Type":"ContainerStarted","Data":"894f78214b39f94ffc7bdd0841e4e95f6c0dab0d20280aa6a65e947f43129ecb"} Oct 30 17:20:53.925229 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:53.925047 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" event={"ID":"ce173a53-c21d-4b94-a7ec-d688705efe9e","Type":"ContainerStarted","Data":"b171c61dd9adaf3c3ad622b98b186677f64c0a3950e51069a854b7e478ac0361"} Oct 30 17:20:53.969754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:53.969710 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:20:54.993980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:54.993620 4828 patch_prober.go:29] interesting pod/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:20:54.993980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:54.993683 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" podUID="ce173a53-c21d-4b94-a7ec-d688705efe9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:20:55.798934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:55.798111 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:55.984925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:55.973281 4828 patch_prober.go:29] interesting pod/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:20:55.984925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:55.973343 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" podUID="ce173a53-c21d-4b94-a7ec-d688705efe9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:20:56.984925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:56.973347 4828 patch_prober.go:29] interesting pod/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:20:56.984925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:56.973414 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" podUID="ce173a53-c21d-4b94-a7ec-d688705efe9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:20:57.202929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.196321 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" podUID="e14ba43d-a967-4948-9b5e-9258ed521f08" containerName="main" containerID="cri-o://eacd7f3e6563deda5b4f9776fb6d437be6e4d8fc25ffda68be94d7f2057596e3" gracePeriod=45 Oct 30 17:20:57.245940 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.245805 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" podUID="e14ba43d-a967-4948-9b5e-9258ed521f08" containerName="main" probeResult="failure" output="Get \"https://10.132.0.174:8000/ready\": dial tcp 10.132.0.174:8000: connect: connection refused" Oct 30 17:20:57.561489 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.561447 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:57.754921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.754461 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:20:57.808980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.808121 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-home\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-home\") pod \"e14ba43d-a967-4948-9b5e-9258ed521f08\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " Oct 30 17:20:57.808980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.808204 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-tmp-dir\") pod \"e14ba43d-a967-4948-9b5e-9258ed521f08\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " Oct 30 17:20:57.808980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.808298 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-model-cache\") pod \"e14ba43d-a967-4948-9b5e-9258ed521f08\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " Oct 30 17:20:57.808980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.808364 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14ba43d-a967-4948-9b5e-9258ed521f08-kube-api-access-rv2zr\" (UniqueName: \"kubernetes.io/projected/e14ba43d-a967-4948-9b5e-9258ed521f08-kube-api-access-rv2zr\") pod \"e14ba43d-a967-4948-9b5e-9258ed521f08\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " Oct 30 17:20:57.808980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.808428 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-dshm\") pod \"e14ba43d-a967-4948-9b5e-9258ed521f08\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " Oct 30 17:20:57.808980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.808587 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e14ba43d-a967-4948-9b5e-9258ed521f08-tls-certs\" (UniqueName: \"kubernetes.io/secret/e14ba43d-a967-4948-9b5e-9258ed521f08-tls-certs\") pod \"e14ba43d-a967-4948-9b5e-9258ed521f08\" (UID: \"e14ba43d-a967-4948-9b5e-9258ed521f08\") " Oct 30 17:20:57.817939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.817849 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-model-cache" pod "e14ba43d-a967-4948-9b5e-9258ed521f08" (UID: "e14ba43d-a967-4948-9b5e-9258ed521f08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:57.819269 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.818238 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-tmp-dir" pod "e14ba43d-a967-4948-9b5e-9258ed521f08" (UID: "e14ba43d-a967-4948-9b5e-9258ed521f08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:57.819786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.819751 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-home" pod "e14ba43d-a967-4948-9b5e-9258ed521f08" (UID: "e14ba43d-a967-4948-9b5e-9258ed521f08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:57.825720 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.825145 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14ba43d-a967-4948-9b5e-9258ed521f08-kube-api-access-rv2zr" pod "e14ba43d-a967-4948-9b5e-9258ed521f08" (UID: "e14ba43d-a967-4948-9b5e-9258ed521f08"). InnerVolumeSpecName "kube-api-access-rv2zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:57.825720 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.825529 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-dshm" pod "e14ba43d-a967-4948-9b5e-9258ed521f08" (UID: "e14ba43d-a967-4948-9b5e-9258ed521f08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:57.831074 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.831035 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e14ba43d-a967-4948-9b5e-9258ed521f08-tls-certs" pod "e14ba43d-a967-4948-9b5e-9258ed521f08" (UID: "e14ba43d-a967-4948-9b5e-9258ed521f08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:20:57.843839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.841662 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" podStartSLOduration=5.841643177 podStartE2EDuration="5.841643177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:53.998304912 +0000 UTC m=+2031.643815093" watchObservedRunningTime="2026-10-30 17:20:57.841643177 +0000 UTC m=+2035.487153357" Oct 30 17:20:57.896876 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.895061 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:57.942431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.912706 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:57.943143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.942768 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv2zr\" (UniqueName: \"kubernetes.io/projected/e14ba43d-a967-4948-9b5e-9258ed521f08-kube-api-access-rv2zr\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:57.943143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.942960 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:57.943143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.942986 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14ba43d-a967-4948-9b5e-9258ed521f08-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:57.943143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.943049 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:57.943143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:57.943065 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14ba43d-a967-4948-9b5e-9258ed521f08-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:20:58.134816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.133086 4828 patch_prober.go:29] interesting pod/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:20:58.134816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.133151 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" podUID="ce173a53-c21d-4b94-a7ec-d688705efe9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:20:58.152018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.145933 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:58.173935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.168755 4828 generic.go:356] "Generic (PLEG): container finished" podID="e14ba43d-a967-4948-9b5e-9258ed521f08" containerID="eacd7f3e6563deda5b4f9776fb6d437be6e4d8fc25ffda68be94d7f2057596e3" exitCode=0 Oct 30 17:20:58.189004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.174274 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" event={"ID":"e14ba43d-a967-4948-9b5e-9258ed521f08","Type":"ContainerDied","Data":"eacd7f3e6563deda5b4f9776fb6d437be6e4d8fc25ffda68be94d7f2057596e3"} Oct 30 17:20:58.189004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.174338 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" event={"ID":"e14ba43d-a967-4948-9b5e-9258ed521f08","Type":"ContainerDied","Data":"bceb46c183563be2f5e2a6136a1f21556325750132afe37b162e77cb3ca46f4f"} Oct 30 17:20:58.189004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.174365 4828 scope.go:122] "RemoveContainer" containerID="eacd7f3e6563deda5b4f9776fb6d437be6e4d8fc25ffda68be94d7f2057596e3" Oct 30 17:20:58.189004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.174577 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5" Oct 30 17:20:58.245995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.245679 4828 scope.go:122] "RemoveContainer" containerID="eacd7f3e6563deda5b4f9776fb6d437be6e4d8fc25ffda68be94d7f2057596e3" Oct 30 17:20:58.246295 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:20:58.246261 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eacd7f3e6563deda5b4f9776fb6d437be6e4d8fc25ffda68be94d7f2057596e3\": container with ID starting with eacd7f3e6563deda5b4f9776fb6d437be6e4d8fc25ffda68be94d7f2057596e3 not found: ID does not exist" containerID="eacd7f3e6563deda5b4f9776fb6d437be6e4d8fc25ffda68be94d7f2057596e3" Oct 30 17:20:58.246382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.246335 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eacd7f3e6563deda5b4f9776fb6d437be6e4d8fc25ffda68be94d7f2057596e3"} err="failed to get container status \"eacd7f3e6563deda5b4f9776fb6d437be6e4d8fc25ffda68be94d7f2057596e3\": rpc error: code = NotFound desc = could not find container \"eacd7f3e6563deda5b4f9776fb6d437be6e4d8fc25ffda68be94d7f2057596e3\": container with ID starting with eacd7f3e6563deda5b4f9776fb6d437be6e4d8fc25ffda68be94d7f2057596e3 not found: ID does not exist" Oct 30 17:20:58.315923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.313433 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5"] Oct 30 17:20:58.319420 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.318372 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:58.319420 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.319266 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f42baf/test-model-f42baf-kserve-576f78b8df-65ws5"] Oct 30 17:20:58.522323 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.519417 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14ba43d-a967-4948-9b5e-9258ed521f08" path="/var/lib/kubelet/pods/e14ba43d-a967-4948-9b5e-9258ed521f08/volumes" Oct 30 17:20:58.974988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.974072 4828 patch_prober.go:29] interesting pod/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:20:58.974988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:58.974132 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" podUID="ce173a53-c21d-4b94-a7ec-d688705efe9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:20:59.702928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:59.702601 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:20:59.970703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:59.970054 4828 patch_prober.go:29] interesting pod/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 17:20:59.970703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:20:59.970116 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" podUID="ce173a53-c21d-4b94-a7ec-d688705efe9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 17:21:00.975362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:00.975327 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:21:00.975885 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:00.975866 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:21:00.976620 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:00.976602 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:21:01.264476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:01.264432 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:21:01.295430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:01.295395 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:21:01.650267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:01.650160 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:02.270722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:02.270688 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:21:03.288752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:03.288713 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:04.939152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:04.939106 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:06.644182 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:06.644141 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:07.914007 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:21:07.913960 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:21:08.235214 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:08.235126 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:08.384786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:08.384749 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:10.071690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:10.070087 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:11.761347 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:11.761305 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:11.945925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:11.934526 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:12.841068 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:12.841028 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:13.897584 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:13.897540 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:14.116675 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:14.116635 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:14.270780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:14.270744 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:15.565973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:15.565934 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:17.141667 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:17.133430 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:17.924358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:17.924306 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:18.082534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:18.082497 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:18.755808 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:18.755765 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:20.434285 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:20.434242 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:21.993814 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:21.993763 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:22.141865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:22.141823 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs"] Oct 30 17:21:22.317430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:22.317385 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:22.480255 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:22.480224 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:23.735039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:23.735003 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:25.037718 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.037681 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs"] Oct 30 17:21:25.038271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.037919 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" podUID="3eb31573-e163-4192-9c08-e582a0bad819" containerName="maas-api" containerID="cri-o://d45c3802188dfe66bcbae338f1ee380d282d7acd685070ea9294692bff4f1fcd" gracePeriod=30 Oct 30 17:21:25.148133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.148092 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:25.525461 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.522272 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:25.761228 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.761189 4828 generic.go:356] "Generic (PLEG): container finished" podID="3eb31573-e163-4192-9c08-e582a0bad819" containerID="d45c3802188dfe66bcbae338f1ee380d282d7acd685070ea9294692bff4f1fcd" exitCode=0 Oct 30 17:21:25.761495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.761461 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" event={"ID":"3eb31573-e163-4192-9c08-e582a0bad819","Type":"ContainerDied","Data":"d45c3802188dfe66bcbae338f1ee380d282d7acd685070ea9294692bff4f1fcd"} Oct 30 17:21:25.762357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.762324 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:25.799767 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.799733 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:21:25.919057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.919005 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:25.976093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.976056 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb31573-e163-4192-9c08-e582a0bad819-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb31573-e163-4192-9c08-e582a0bad819-openshift-service-ca\") pod \"3eb31573-e163-4192-9c08-e582a0bad819\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " Oct 30 17:21:25.976289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.976133 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb31573-e163-4192-9c08-e582a0bad819-kube-api-access-x59rv\" (UniqueName: \"kubernetes.io/projected/3eb31573-e163-4192-9c08-e582a0bad819-kube-api-access-x59rv\") pod \"3eb31573-e163-4192-9c08-e582a0bad819\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " Oct 30 17:21:25.976289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.976161 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-maas-api-tls\") pod \"3eb31573-e163-4192-9c08-e582a0bad819\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " Oct 30 17:21:25.976516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.976490 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-metrics-certs\") pod \"3eb31573-e163-4192-9c08-e582a0bad819\" (UID: \"3eb31573-e163-4192-9c08-e582a0bad819\") " Oct 30 17:21:25.976877 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.976853 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb31573-e163-4192-9c08-e582a0bad819-openshift-service-ca" pod "3eb31573-e163-4192-9c08-e582a0bad819" (UID: "3eb31573-e163-4192-9c08-e582a0bad819"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:25.977327 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.977306 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb31573-e163-4192-9c08-e582a0bad819-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:25.980623 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.980420 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb31573-e163-4192-9c08-e582a0bad819-kube-api-access-x59rv" pod "3eb31573-e163-4192-9c08-e582a0bad819" (UID: "3eb31573-e163-4192-9c08-e582a0bad819"). InnerVolumeSpecName "kube-api-access-x59rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:25.982434 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.982371 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-metrics-certs" pod "3eb31573-e163-4192-9c08-e582a0bad819" (UID: "3eb31573-e163-4192-9c08-e582a0bad819"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:25.982694 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:25.982666 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-maas-api-tls" pod "3eb31573-e163-4192-9c08-e582a0bad819" (UID: "3eb31573-e163-4192-9c08-e582a0bad819"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:26.077036 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:26.076954 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:26.078676 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:26.078652 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:26.078771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:26.078677 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x59rv\" (UniqueName: \"kubernetes.io/projected/3eb31573-e163-4192-9c08-e582a0bad819-kube-api-access-x59rv\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:26.078771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:26.078687 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eb31573-e163-4192-9c08-e582a0bad819-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:26.234686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:26.234635 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:26.629866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:26.629827 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:26.774436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:26.774357 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" Oct 30 17:21:26.774691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:26.774390 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs" event={"ID":"3eb31573-e163-4192-9c08-e582a0bad819","Type":"ContainerDied","Data":"f1602ab5c83bd0dd6c70d9338a0d4aa4be8ffc899c895699c1664bde68df3c83"} Oct 30 17:21:26.774848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:26.774832 4828 scope.go:122] "RemoveContainer" containerID="d45c3802188dfe66bcbae338f1ee380d282d7acd685070ea9294692bff4f1fcd" Oct 30 17:21:26.872090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:26.872051 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs"] Oct 30 17:21:26.888260 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:26.888213 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f42baf-cf768db8d-k9szs"] Oct 30 17:21:27.142771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:27.142659 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" podUID="ea897670-fd5e-45cc-8529-7cbe6a7d6a06" containerName="payload-pre-processing" containerID="cri-o://9e04df38ba8f5c1861084ea111df06772027576f488e41b40c03cfb2088ca0bb" gracePeriod=25 Oct 30 17:21:27.984766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:27.984722 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:28.496068 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:28.496031 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb31573-e163-4192-9c08-e582a0bad819" path="/var/lib/kubelet/pods/3eb31573-e163-4192-9c08-e582a0bad819/volumes" Oct 30 17:21:29.710922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:29.704619 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:31.481161 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:31.481116 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:31.604247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:31.604209 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" podUID="ea897670-fd5e-45cc-8529-7cbe6a7d6a06" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:21:33.061667 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:33.061615 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:33.685526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:33.685488 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:33.847456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:33.847420 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:34.070559 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:34.070523 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98"] Oct 30 17:21:34.071968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:34.071728 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" containerID="cri-o://6569fb90f8f5573fd30307193c2b4d5e08708e90df67e9fa87955f97bbefcbd3" gracePeriod=30 Oct 30 17:21:34.085515 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:34.085481 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc"] Oct 30 17:21:34.085760 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:34.085737 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" podUID="fc599032-c154-45cb-89cd-bebabe6c45e7" containerName="istio-proxy" containerID="cri-o://cb0e1b05afb7f4d6813ded9ae657360a856b4512ce3541d7eba37e2ed13b6abc" gracePeriod=30 Oct 30 17:21:34.148075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:34.148039 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:34.960772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:34.960736 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:35.435592 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:35.435549 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:21:35.436124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:35.435622 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:21:36.363573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:36.363533 4828 patch_prober.go:29] interesting pod/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:21:36.363767 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:36.363610 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" podUID="fc599032-c154-45cb-89cd-bebabe6c45e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:21:37.007969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:37.007529 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:37.914583 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:21:37.914536 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:21:39.379061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.378938 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:39.479151 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.473744 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:21:39.548792 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.548761 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:21:39.569661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.569623 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-socket\") pod \"fc599032-c154-45cb-89cd-bebabe6c45e7\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " Oct 30 17:21:39.569861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.569716 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-kube-api-access-w9z7m\" (UniqueName: \"kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-kube-api-access-w9z7m\") pod \"fc599032-c154-45cb-89cd-bebabe6c45e7\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " Oct 30 17:21:39.569861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.569761 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-envoy\") pod \"fc599032-c154-45cb-89cd-bebabe6c45e7\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " Oct 30 17:21:39.569861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.569796 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-kube-api-access-mcbjk\" (UniqueName: \"kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-kube-api-access-mcbjk\") pod \"300e1607-7771-4378-b40a-56cf985f25d5\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " Oct 30 17:21:39.569861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.569823 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-certs\") pod \"300e1607-7771-4378-b40a-56cf985f25d5\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " Oct 30 17:21:39.569861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.569855 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc599032-c154-45cb-89cd-bebabe6c45e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc599032-c154-45cb-89cd-bebabe6c45e7-istiod-ca-cert\") pod \"fc599032-c154-45cb-89cd-bebabe6c45e7\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " Oct 30 17:21:39.570234 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570208 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc599032-c154-45cb-89cd-bebabe6c45e7-istiod-ca-cert" pod "fc599032-c154-45cb-89cd-bebabe6c45e7" (UID: "fc599032-c154-45cb-89cd-bebabe6c45e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:39.570298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570213 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-envoy\") pod \"300e1607-7771-4378-b40a-56cf985f25d5\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " Oct 30 17:21:39.570383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570363 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-socket" pod "fc599032-c154-45cb-89cd-bebabe6c45e7" (UID: "fc599032-c154-45cb-89cd-bebabe6c45e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.570443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570421 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-credential-socket\") pod \"300e1607-7771-4378-b40a-56cf985f25d5\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " Oct 30 17:21:39.570509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570491 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-socket\") pod \"300e1607-7771-4378-b40a-56cf985f25d5\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " Oct 30 17:21:39.570576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570553 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-data\") pod \"fc599032-c154-45cb-89cd-bebabe6c45e7\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " Oct 30 17:21:39.570631 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570588 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/300e1607-7771-4378-b40a-56cf985f25d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300e1607-7771-4378-b40a-56cf985f25d5-istiod-ca-cert\") pod \"300e1607-7771-4378-b40a-56cf985f25d5\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " Oct 30 17:21:39.570688 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570635 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-data\") pod \"300e1607-7771-4378-b40a-56cf985f25d5\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " Oct 30 17:21:39.570740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570723 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/300e1607-7771-4378-b40a-56cf985f25d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300e1607-7771-4378-b40a-56cf985f25d5-istio-podinfo\") pod \"300e1607-7771-4378-b40a-56cf985f25d5\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " Oct 30 17:21:39.570796 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570772 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-credential-socket\") pod \"fc599032-c154-45cb-89cd-bebabe6c45e7\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " Oct 30 17:21:39.570859 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570791 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-podinfo\") pod \"fc599032-c154-45cb-89cd-bebabe6c45e7\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " Oct 30 17:21:39.570859 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570845 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-token\" (UniqueName: \"kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-token\") pod \"fc599032-c154-45cb-89cd-bebabe6c45e7\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " Oct 30 17:21:39.571022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570941 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-data" pod "fc599032-c154-45cb-89cd-bebabe6c45e7" (UID: "fc599032-c154-45cb-89cd-bebabe6c45e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.571022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.570976 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-istio-token\" (UniqueName: \"kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-istio-token\") pod \"300e1607-7771-4378-b40a-56cf985f25d5\" (UID: \"300e1607-7771-4378-b40a-56cf985f25d5\") " Oct 30 17:21:39.571022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.571004 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-certs\") pod \"fc599032-c154-45cb-89cd-bebabe6c45e7\" (UID: \"fc599032-c154-45cb-89cd-bebabe6c45e7\") " Oct 30 17:21:39.571196 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.571145 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-credential-socket" pod "300e1607-7771-4378-b40a-56cf985f25d5" (UID: "300e1607-7771-4378-b40a-56cf985f25d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.571348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.571324 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-socket" pod "300e1607-7771-4378-b40a-56cf985f25d5" (UID: "300e1607-7771-4378-b40a-56cf985f25d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.571885 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.571860 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc599032-c154-45cb-89cd-bebabe6c45e7-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.572034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.571890 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.572034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.571921 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.572034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.571936 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.572034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.571950 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.572334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.572311 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-credential-socket" pod "fc599032-c154-45cb-89cd-bebabe6c45e7" (UID: "fc599032-c154-45cb-89cd-bebabe6c45e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.572651 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.572628 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300e1607-7771-4378-b40a-56cf985f25d5-istiod-ca-cert" pod "300e1607-7771-4378-b40a-56cf985f25d5" (UID: "300e1607-7771-4378-b40a-56cf985f25d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:39.572890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.572870 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-data" pod "300e1607-7771-4378-b40a-56cf985f25d5" (UID: "300e1607-7771-4378-b40a-56cf985f25d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.573566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.573543 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-podinfo" pod "fc599032-c154-45cb-89cd-bebabe6c45e7" (UID: "fc599032-c154-45cb-89cd-bebabe6c45e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:39.574128 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.574102 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-certs" pod "fc599032-c154-45cb-89cd-bebabe6c45e7" (UID: "fc599032-c154-45cb-89cd-bebabe6c45e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.574589 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.574565 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-certs" pod "300e1607-7771-4378-b40a-56cf985f25d5" (UID: "300e1607-7771-4378-b40a-56cf985f25d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.575350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.575324 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/300e1607-7771-4378-b40a-56cf985f25d5-istio-podinfo" pod "300e1607-7771-4378-b40a-56cf985f25d5" (UID: "300e1607-7771-4378-b40a-56cf985f25d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:39.576031 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.576000 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-token" pod "fc599032-c154-45cb-89cd-bebabe6c45e7" (UID: "fc599032-c154-45cb-89cd-bebabe6c45e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:39.576616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.576591 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-istio-token" pod "300e1607-7771-4378-b40a-56cf985f25d5" (UID: "300e1607-7771-4378-b40a-56cf985f25d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:39.580415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.580221 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-kube-api-access-w9z7m" pod "fc599032-c154-45cb-89cd-bebabe6c45e7" (UID: "fc599032-c154-45cb-89cd-bebabe6c45e7"). InnerVolumeSpecName "kube-api-access-w9z7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:39.580415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.580226 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-envoy" pod "300e1607-7771-4378-b40a-56cf985f25d5" (UID: "300e1607-7771-4378-b40a-56cf985f25d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.580415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.580276 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-kube-api-access-mcbjk" pod "300e1607-7771-4378-b40a-56cf985f25d5" (UID: "300e1607-7771-4378-b40a-56cf985f25d5"). InnerVolumeSpecName "kube-api-access-mcbjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:39.582004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.581980 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-envoy" pod "fc599032-c154-45cb-89cd-bebabe6c45e7" (UID: "fc599032-c154-45cb-89cd-bebabe6c45e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:39.673655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.673560 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300e1607-7771-4378-b40a-56cf985f25d5-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.673655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.673592 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.673655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.673607 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300e1607-7771-4378-b40a-56cf985f25d5-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.673655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.673619 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.673655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.673635 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.673655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.673646 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.673655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.673658 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.674082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.673672 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.674082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.673684 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9z7m\" (UniqueName: \"kubernetes.io/projected/fc599032-c154-45cb-89cd-bebabe6c45e7-kube-api-access-w9z7m\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.674082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.673696 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc599032-c154-45cb-89cd-bebabe6c45e7-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.674082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.673707 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcbjk\" (UniqueName: \"kubernetes.io/projected/300e1607-7771-4378-b40a-56cf985f25d5-kube-api-access-mcbjk\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.674082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.673718 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.674082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.673737 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300e1607-7771-4378-b40a-56cf985f25d5-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:39.909782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.909746 4828 generic.go:356] "Generic (PLEG): container finished" podID="fc599032-c154-45cb-89cd-bebabe6c45e7" containerID="cb0e1b05afb7f4d6813ded9ae657360a856b4512ce3541d7eba37e2ed13b6abc" exitCode=0 Oct 30 17:21:39.910007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.909798 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" Oct 30 17:21:39.910007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.909838 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" event={"ID":"fc599032-c154-45cb-89cd-bebabe6c45e7","Type":"ContainerDied","Data":"cb0e1b05afb7f4d6813ded9ae657360a856b4512ce3541d7eba37e2ed13b6abc"} Oct 30 17:21:39.910007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.909942 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc" event={"ID":"fc599032-c154-45cb-89cd-bebabe6c45e7","Type":"ContainerDied","Data":"871e37f74be1bdf45b8796604ac176a61fcadc13e3a8158edf63d4c6b32e6d7d"} Oct 30 17:21:39.910007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.909962 4828 scope.go:122] "RemoveContainer" containerID="cb0e1b05afb7f4d6813ded9ae657360a856b4512ce3541d7eba37e2ed13b6abc" Oct 30 17:21:39.911999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.911888 4828 generic.go:356] "Generic (PLEG): container finished" podID="300e1607-7771-4378-b40a-56cf985f25d5" containerID="6569fb90f8f5573fd30307193c2b4d5e08708e90df67e9fa87955f97bbefcbd3" exitCode=0 Oct 30 17:21:39.912098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.912002 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" Oct 30 17:21:39.912098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.911958 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" event={"ID":"300e1607-7771-4378-b40a-56cf985f25d5","Type":"ContainerDied","Data":"6569fb90f8f5573fd30307193c2b4d5e08708e90df67e9fa87955f97bbefcbd3"} Oct 30 17:21:39.912178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.912110 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98" event={"ID":"300e1607-7771-4378-b40a-56cf985f25d5","Type":"ContainerDied","Data":"7669a00dafec1a23e1028fa176a4181f33ea5c87c485a78d2bfbf131b635d625"} Oct 30 17:21:39.926258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.926241 4828 scope.go:122] "RemoveContainer" containerID="cb0e1b05afb7f4d6813ded9ae657360a856b4512ce3541d7eba37e2ed13b6abc" Oct 30 17:21:39.926552 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:21:39.926533 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0e1b05afb7f4d6813ded9ae657360a856b4512ce3541d7eba37e2ed13b6abc\": container with ID starting with cb0e1b05afb7f4d6813ded9ae657360a856b4512ce3541d7eba37e2ed13b6abc not found: ID does not exist" containerID="cb0e1b05afb7f4d6813ded9ae657360a856b4512ce3541d7eba37e2ed13b6abc" Oct 30 17:21:39.926627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.926565 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0e1b05afb7f4d6813ded9ae657360a856b4512ce3541d7eba37e2ed13b6abc"} err="failed to get container status \"cb0e1b05afb7f4d6813ded9ae657360a856b4512ce3541d7eba37e2ed13b6abc\": rpc error: code = NotFound desc = could not find container \"cb0e1b05afb7f4d6813ded9ae657360a856b4512ce3541d7eba37e2ed13b6abc\": container with ID starting with cb0e1b05afb7f4d6813ded9ae657360a856b4512ce3541d7eba37e2ed13b6abc not found: ID does not exist" Oct 30 17:21:39.926627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.926589 4828 scope.go:122] "RemoveContainer" containerID="6569fb90f8f5573fd30307193c2b4d5e08708e90df67e9fa87955f97bbefcbd3" Oct 30 17:21:39.937348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.937331 4828 scope.go:122] "RemoveContainer" containerID="6ab43b9e3392c1afa54e24d89e2cdf67e66ade2420a9e7c8bf9588f3e481fd95" Oct 30 17:21:39.948237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.948216 4828 scope.go:122] "RemoveContainer" containerID="6569fb90f8f5573fd30307193c2b4d5e08708e90df67e9fa87955f97bbefcbd3" Oct 30 17:21:39.948561 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:21:39.948536 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6569fb90f8f5573fd30307193c2b4d5e08708e90df67e9fa87955f97bbefcbd3\": container with ID starting with 6569fb90f8f5573fd30307193c2b4d5e08708e90df67e9fa87955f97bbefcbd3 not found: ID does not exist" containerID="6569fb90f8f5573fd30307193c2b4d5e08708e90df67e9fa87955f97bbefcbd3" Oct 30 17:21:39.948642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.948577 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6569fb90f8f5573fd30307193c2b4d5e08708e90df67e9fa87955f97bbefcbd3"} err="failed to get container status \"6569fb90f8f5573fd30307193c2b4d5e08708e90df67e9fa87955f97bbefcbd3\": rpc error: code = NotFound desc = could not find container \"6569fb90f8f5573fd30307193c2b4d5e08708e90df67e9fa87955f97bbefcbd3\": container with ID starting with 6569fb90f8f5573fd30307193c2b4d5e08708e90df67e9fa87955f97bbefcbd3 not found: ID does not exist" Oct 30 17:21:39.948642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.948594 4828 scope.go:122] "RemoveContainer" containerID="6ab43b9e3392c1afa54e24d89e2cdf67e66ade2420a9e7c8bf9588f3e481fd95" Oct 30 17:21:39.948858 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:21:39.948834 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab43b9e3392c1afa54e24d89e2cdf67e66ade2420a9e7c8bf9588f3e481fd95\": container with ID starting with 6ab43b9e3392c1afa54e24d89e2cdf67e66ade2420a9e7c8bf9588f3e481fd95 not found: ID does not exist" containerID="6ab43b9e3392c1afa54e24d89e2cdf67e66ade2420a9e7c8bf9588f3e481fd95" Oct 30 17:21:39.948958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.948867 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab43b9e3392c1afa54e24d89e2cdf67e66ade2420a9e7c8bf9588f3e481fd95"} err="failed to get container status \"6ab43b9e3392c1afa54e24d89e2cdf67e66ade2420a9e7c8bf9588f3e481fd95\": rpc error: code = NotFound desc = could not find container \"6ab43b9e3392c1afa54e24d89e2cdf67e66ade2420a9e7c8bf9588f3e481fd95\": container with ID starting with 6ab43b9e3392c1afa54e24d89e2cdf67e66ade2420a9e7c8bf9588f3e481fd95 not found: ID does not exist" Oct 30 17:21:39.985295 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.985263 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc"] Oct 30 17:21:39.997793 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:39.997757 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-kp9jc"] Oct 30 17:21:40.025908 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:40.025863 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98"] Oct 30 17:21:40.031206 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:40.031185 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f42baf-openshift-default-8468c49fd5-cmq98"] Oct 30 17:21:40.498779 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:40.498738 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="300e1607-7771-4378-b40a-56cf985f25d5" path="/var/lib/kubelet/pods/300e1607-7771-4378-b40a-56cf985f25d5/volumes" Oct 30 17:21:40.499376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:40.499357 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc599032-c154-45cb-89cd-bebabe6c45e7" path="/var/lib/kubelet/pods/fc599032-c154-45cb-89cd-bebabe6c45e7/volumes" Oct 30 17:21:41.087924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:41.083747 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:41.603181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:41.603139 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" podUID="ea897670-fd5e-45cc-8529-7cbe6a7d6a06" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:21:42.840021 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:42.839964 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:44.463924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:44.461312 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:44.725913 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:44.725816 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt"] Oct 30 17:21:44.870748 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:44.870714 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:45.038773 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:45.038739 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:46.246734 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:46.240585 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:47.476469 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:47.476433 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" Oct 30 17:21:47.571957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:47.571918 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-plugins-config-volume\") pod \"ea897670-fd5e-45cc-8529-7cbe6a7d6a06\" (UID: \"ea897670-fd5e-45cc-8529-7cbe6a7d6a06\") " Oct 30 17:21:47.572329 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:47.572299 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-plugins-config-volume" pod "ea897670-fd5e-45cc-8529-7cbe6a7d6a06" (UID: "ea897670-fd5e-45cc-8529-7cbe6a7d6a06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:47.572649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:47.572620 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-kube-api-access-vxvvk\" (UniqueName: \"kubernetes.io/projected/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-kube-api-access-vxvvk\") pod \"ea897670-fd5e-45cc-8529-7cbe6a7d6a06\" (UID: \"ea897670-fd5e-45cc-8529-7cbe6a7d6a06\") " Oct 30 17:21:47.573776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:47.573750 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:47.587355 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:47.587325 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-kube-api-access-vxvvk" pod "ea897670-fd5e-45cc-8529-7cbe6a7d6a06" (UID: "ea897670-fd5e-45cc-8529-7cbe6a7d6a06"). InnerVolumeSpecName "kube-api-access-vxvvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:47.676030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:47.675940 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxvvk\" (UniqueName: \"kubernetes.io/projected/ea897670-fd5e-45cc-8529-7cbe6a7d6a06-kube-api-access-vxvvk\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:21:47.869947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:47.868137 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:48.042223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:48.042183 4828 generic.go:356] "Generic (PLEG): container finished" podID="ea897670-fd5e-45cc-8529-7cbe6a7d6a06" containerID="9e04df38ba8f5c1861084ea111df06772027576f488e41b40c03cfb2088ca0bb" exitCode=0 Oct 30 17:21:48.042377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:48.042228 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" Oct 30 17:21:48.042377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:48.042267 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" event={"ID":"ea897670-fd5e-45cc-8529-7cbe6a7d6a06","Type":"ContainerDied","Data":"9e04df38ba8f5c1861084ea111df06772027576f488e41b40c03cfb2088ca0bb"} Oct 30 17:21:48.042377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:48.042320 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs" event={"ID":"ea897670-fd5e-45cc-8529-7cbe6a7d6a06","Type":"ContainerDied","Data":"5a4ddc7d6f8a9c9a3a669935ed7e15e998d6a1a286859d81d666939b2dfc37b8"} Oct 30 17:21:48.042377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:48.042343 4828 scope.go:122] "RemoveContainer" containerID="9e04df38ba8f5c1861084ea111df06772027576f488e41b40c03cfb2088ca0bb" Oct 30 17:21:48.054093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:48.054074 4828 scope.go:122] "RemoveContainer" containerID="9e04df38ba8f5c1861084ea111df06772027576f488e41b40c03cfb2088ca0bb" Oct 30 17:21:48.054426 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:21:48.054405 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e04df38ba8f5c1861084ea111df06772027576f488e41b40c03cfb2088ca0bb\": container with ID starting with 9e04df38ba8f5c1861084ea111df06772027576f488e41b40c03cfb2088ca0bb not found: ID does not exist" containerID="9e04df38ba8f5c1861084ea111df06772027576f488e41b40c03cfb2088ca0bb" Oct 30 17:21:48.054493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:48.054439 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e04df38ba8f5c1861084ea111df06772027576f488e41b40c03cfb2088ca0bb"} err="failed to get container status \"9e04df38ba8f5c1861084ea111df06772027576f488e41b40c03cfb2088ca0bb\": rpc error: code = NotFound desc = could not find container \"9e04df38ba8f5c1861084ea111df06772027576f488e41b40c03cfb2088ca0bb\": container with ID starting with 9e04df38ba8f5c1861084ea111df06772027576f488e41b40c03cfb2088ca0bb not found: ID does not exist" Oct 30 17:21:48.099733 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:48.099691 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs"] Oct 30 17:21:48.119919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:48.119861 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d003a5-7865f9c759-dftbs"] Oct 30 17:21:48.496310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:48.496223 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea897670-fd5e-45cc-8529-7cbe6a7d6a06" path="/var/lib/kubelet/pods/ea897670-fd5e-45cc-8529-7cbe6a7d6a06/volumes" Oct 30 17:21:49.551789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:49.551754 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:49.705463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:49.705429 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:49.726389 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:49.726347 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" podUID="403f830a-d924-4d8f-a787-1e43cd9f6a5c" containerName="payload-pre-processing" containerID="cri-o://0510ea6d346522ca30581245cf8cd6fc15d5f3fd72f176d7f61515e7bb700fe2" gracePeriod=25 Oct 30 17:21:51.307160 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:51.307057 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:52.953926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:52.952002 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:54.791928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.790868 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.883134 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd"] Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884063 4828 cpu_manager.go:414] "containerMap: removing container" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884081 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884092 4828 cpu_manager.go:414] "containerMap: removing container" podUID="ea897670-fd5e-45cc-8529-7cbe6a7d6a06" containerName="payload-pre-processing" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884098 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea897670-fd5e-45cc-8529-7cbe6a7d6a06" containerName="payload-pre-processing" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884123 4828 cpu_manager.go:414] "containerMap: removing container" podUID="fc599032-c154-45cb-89cd-bebabe6c45e7" containerName="istio-proxy" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884130 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc599032-c154-45cb-89cd-bebabe6c45e7" containerName="istio-proxy" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884147 4828 cpu_manager.go:414] "containerMap: removing container" podUID="e14ba43d-a967-4948-9b5e-9258ed521f08" containerName="main" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884153 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14ba43d-a967-4948-9b5e-9258ed521f08" containerName="main" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884175 4828 cpu_manager.go:414] "containerMap: removing container" podUID="3eb31573-e163-4192-9c08-e582a0bad819" containerName="maas-api" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884180 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb31573-e163-4192-9c08-e582a0bad819" containerName="maas-api" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884301 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14ba43d-a967-4948-9b5e-9258ed521f08" containerName="main" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884312 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884320 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb31573-e163-4192-9c08-e582a0bad819" containerName="maas-api" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884329 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884336 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea897670-fd5e-45cc-8529-7cbe6a7d6a06" containerName="payload-pre-processing" Oct 30 17:21:54.885017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.884344 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc599032-c154-45cb-89cd-bebabe6c45e7" containerName="istio-proxy" Oct 30 17:21:54.889850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.889823 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:54.922958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.918567 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c45877-openshift-default-dockercfg-29smt\"" Oct 30 17:21:54.927412 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.927378 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd"] Oct 30 17:21:54.996326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.996286 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-podinfo\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:54.996538 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.996352 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-credential-socket\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:54.996538 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.996372 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/875c10a1-3b01-46e0-a4ed-62e65c551a91-istiod-ca-cert\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:54.996538 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.996511 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6xrf\" (UniqueName: \"kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-kube-api-access-k6xrf\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:54.996775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.996632 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-certs\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:54.996775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.996672 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-data\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:54.996845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.996821 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-token\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:54.996942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.996926 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-socket\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:54.997011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:54.996998 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-envoy\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.099468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.099419 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-podinfo\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.099673 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.099508 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-credential-socket\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.099673 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.099535 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/875c10a1-3b01-46e0-a4ed-62e65c551a91-istiod-ca-cert\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.099673 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.099579 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6xrf\" (UniqueName: \"kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-kube-api-access-k6xrf\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.099853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.099685 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-certs\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.099853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.099713 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-data\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.099853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.099791 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-token\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.099853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.099848 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-socket\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.100120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.099976 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-envoy\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.100508 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.100304 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-data\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.100508 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.100397 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-credential-socket\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.100508 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.100414 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/875c10a1-3b01-46e0-a4ed-62e65c551a91-istiod-ca-cert\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.100508 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.100471 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-certs\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.100749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.100735 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-socket\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.102834 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.102800 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-envoy\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.103017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.102993 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-podinfo\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.124403 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.124373 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-token\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.124534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.124442 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6xrf\" (UniqueName: \"kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-kube-api-access-k6xrf\") pod \"e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.211582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.211477 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:55.547949 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:21:55.547920 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875c10a1_3b01_46e0_a4ed_62e65c551a91.slice/crio-92c42d63db83d2746d4100d3dd56073dbd05b98a2747bc805b43e918cf73a19e WatchSource:0}: Error finding container 92c42d63db83d2746d4100d3dd56073dbd05b98a2747bc805b43e918cf73a19e: Status 404 returned error can't find the container with id 92c42d63db83d2746d4100d3dd56073dbd05b98a2747bc805b43e918cf73a19e Oct 30 17:21:55.549388 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.549362 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd"] Oct 30 17:21:55.550985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.550950 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:21:55.551051 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.551018 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:21:55.551095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:55.551048 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:21:56.120640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:56.120604 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" event={"ID":"875c10a1-3b01-46e0-a4ed-62e65c551a91","Type":"ContainerStarted","Data":"aaa0079c18457abb10c352ffe2a134f82b91761c5c205504a3f55b42f24503f4"} Oct 30 17:21:56.121072 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:56.120677 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" event={"ID":"875c10a1-3b01-46e0-a4ed-62e65c551a91","Type":"ContainerStarted","Data":"92c42d63db83d2746d4100d3dd56073dbd05b98a2747bc805b43e918cf73a19e"} Oct 30 17:21:56.173507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:56.173440 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" podStartSLOduration=2.173417341 podStartE2EDuration="2.173417341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:56.168152301 +0000 UTC m=+2093.813662489" watchObservedRunningTime="2026-10-30 17:21:56.173417341 +0000 UTC m=+2093.818927522" Oct 30 17:21:56.212534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:56.212496 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:21:56.214576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:56.214545 4828 patch_prober.go:29] interesting pod/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 17:21:56.214871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:56.214841 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" podUID="875c10a1-3b01-46e0-a4ed-62e65c551a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 17:21:56.584966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:56.558281 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:57.213488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:57.213004 4828 patch_prober.go:29] interesting pod/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 17:21:57.213488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:57.213184 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" podUID="875c10a1-3b01-46e0-a4ed-62e65c551a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 17:21:58.072924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:58.070462 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:58.212166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:58.212123 4828 patch_prober.go:29] interesting pod/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 17:21:58.212356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:58.212214 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" podUID="875c10a1-3b01-46e0-a4ed-62e65c551a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 17:21:58.239408 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:58.239372 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:58.402118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:58.402015 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" podUID="403f830a-d924-4d8f-a787-1e43cd9f6a5c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:21:58.536272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:58.536223 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:21:59.212729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:59.212689 4828 patch_prober.go:29] interesting pod/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 17:21:59.212950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:21:59.212757 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" podUID="875c10a1-3b01-46e0-a4ed-62e65c551a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 17:22:00.218922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:00.216208 4828 patch_prober.go:29] interesting pod/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 17:22:00.218922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:00.216274 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" podUID="875c10a1-3b01-46e0-a4ed-62e65c551a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 17:22:00.218922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:00.217141 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:01.212252 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:01.212205 4828 patch_prober.go:29] interesting pod/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 17:22:01.212479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:01.212277 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" podUID="875c10a1-3b01-46e0-a4ed-62e65c551a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 17:22:01.940925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:01.939518 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:02.211932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:02.211843 4828 patch_prober.go:29] interesting pod/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 30 17:22:02.212088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:02.212005 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" podUID="875c10a1-3b01-46e0-a4ed-62e65c551a91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 30 17:22:03.217066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:03.217034 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:22:03.217566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:03.217427 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:22:03.218826 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:03.218805 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:22:03.521969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:03.521923 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:04.987922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:04.985025 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:05.207413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:05.207365 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt"] Oct 30 17:22:05.207740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:05.207687 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" podUID="c6cb4e0e-750c-4846-ab7c-375ac9fbe956" containerName="maas-api" containerID="cri-o://fa48fdd1a3858e09c46f254bc77fd2ecb3018d24d2cca5327d59bf009d774445" gracePeriod=30 Oct 30 17:22:05.234012 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:05.227617 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:06.048186 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.048154 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:06.075919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.075876 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:22:06.121843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.119833 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:22:06.137500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.136782 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:22:06.142067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.140198 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-metrics-certs\") pod \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " Oct 30 17:22:06.142067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.140318 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-kube-api-access-frwsv\" (UniqueName: \"kubernetes.io/projected/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-kube-api-access-frwsv\") pod \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " Oct 30 17:22:06.142067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.140377 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-maas-api-tls\") pod \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " Oct 30 17:22:06.142067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.140616 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-openshift-service-ca\") pod \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\" (UID: \"c6cb4e0e-750c-4846-ab7c-375ac9fbe956\") " Oct 30 17:22:06.142067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.141444 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-openshift-service-ca" pod "c6cb4e0e-750c-4846-ab7c-375ac9fbe956" (UID: "c6cb4e0e-750c-4846-ab7c-375ac9fbe956"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:06.149411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.146608 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-metrics-certs" pod "c6cb4e0e-750c-4846-ab7c-375ac9fbe956" (UID: "c6cb4e0e-750c-4846-ab7c-375ac9fbe956"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:06.149411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.147011 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-kube-api-access-frwsv" pod "c6cb4e0e-750c-4846-ab7c-375ac9fbe956" (UID: "c6cb4e0e-750c-4846-ab7c-375ac9fbe956"). InnerVolumeSpecName "kube-api-access-frwsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:06.149411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.148248 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-maas-api-tls" pod "c6cb4e0e-750c-4846-ab7c-375ac9fbe956" (UID: "c6cb4e0e-750c-4846-ab7c-375ac9fbe956"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:06.200957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.200891 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:06.231645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.231606 4828 generic.go:356] "Generic (PLEG): container finished" podID="c6cb4e0e-750c-4846-ab7c-375ac9fbe956" containerID="fa48fdd1a3858e09c46f254bc77fd2ecb3018d24d2cca5327d59bf009d774445" exitCode=0 Oct 30 17:22:06.231846 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.231648 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" Oct 30 17:22:06.231846 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.231666 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" event={"ID":"c6cb4e0e-750c-4846-ab7c-375ac9fbe956","Type":"ContainerDied","Data":"fa48fdd1a3858e09c46f254bc77fd2ecb3018d24d2cca5327d59bf009d774445"} Oct 30 17:22:06.231846 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.231721 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt" event={"ID":"c6cb4e0e-750c-4846-ab7c-375ac9fbe956","Type":"ContainerDied","Data":"a0c8d7611c988e7757cef7071182b14d91b59d89243dd67c1062129a4271322a"} Oct 30 17:22:06.231846 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.231750 4828 scope.go:122] "RemoveContainer" containerID="fa48fdd1a3858e09c46f254bc77fd2ecb3018d24d2cca5327d59bf009d774445" Oct 30 17:22:06.242887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.242858 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:06.243060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.242892 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frwsv\" (UniqueName: \"kubernetes.io/projected/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-kube-api-access-frwsv\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:06.243060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.242925 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:06.243060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.242941 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6cb4e0e-750c-4846-ab7c-375ac9fbe956-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:06.247998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.247973 4828 scope.go:122] "RemoveContainer" containerID="fa48fdd1a3858e09c46f254bc77fd2ecb3018d24d2cca5327d59bf009d774445" Oct 30 17:22:06.248310 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:22:06.248289 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa48fdd1a3858e09c46f254bc77fd2ecb3018d24d2cca5327d59bf009d774445\": container with ID starting with fa48fdd1a3858e09c46f254bc77fd2ecb3018d24d2cca5327d59bf009d774445 not found: ID does not exist" containerID="fa48fdd1a3858e09c46f254bc77fd2ecb3018d24d2cca5327d59bf009d774445" Oct 30 17:22:06.248381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.248322 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa48fdd1a3858e09c46f254bc77fd2ecb3018d24d2cca5327d59bf009d774445"} err="failed to get container status \"fa48fdd1a3858e09c46f254bc77fd2ecb3018d24d2cca5327d59bf009d774445\": rpc error: code = NotFound desc = could not find container \"fa48fdd1a3858e09c46f254bc77fd2ecb3018d24d2cca5327d59bf009d774445\": container with ID starting with fa48fdd1a3858e09c46f254bc77fd2ecb3018d24d2cca5327d59bf009d774445 not found: ID does not exist" Oct 30 17:22:06.290082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.290027 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt"] Oct 30 17:22:06.294842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.294799 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d003a5-755b5fbfd6-qsgwt"] Oct 30 17:22:06.390754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.390697 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:06.494722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.494638 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6cb4e0e-750c-4846-ab7c-375ac9fbe956" path="/var/lib/kubelet/pods/c6cb4e0e-750c-4846-ab7c-375ac9fbe956/volumes" Oct 30 17:22:06.738352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:06.738316 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:07.912797 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:22:07.912754 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:22:08.142925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:08.141501 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:08.380124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:08.380078 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" podUID="403f830a-d924-4d8f-a787-1e43cd9f6a5c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:22:09.808005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:09.807827 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:10.078648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.078619 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" Oct 30 17:22:10.141169 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.141131 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:10.218345 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.218311 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403f830a-d924-4d8f-a787-1e43cd9f6a5c-kube-api-access-pfn8z\" (UniqueName: \"kubernetes.io/projected/403f830a-d924-4d8f-a787-1e43cd9f6a5c-kube-api-access-pfn8z\") pod \"403f830a-d924-4d8f-a787-1e43cd9f6a5c\" (UID: \"403f830a-d924-4d8f-a787-1e43cd9f6a5c\") " Oct 30 17:22:10.218552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.218426 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403f830a-d924-4d8f-a787-1e43cd9f6a5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/403f830a-d924-4d8f-a787-1e43cd9f6a5c-plugins-config-volume\") pod \"403f830a-d924-4d8f-a787-1e43cd9f6a5c\" (UID: \"403f830a-d924-4d8f-a787-1e43cd9f6a5c\") " Oct 30 17:22:10.218888 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.218863 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403f830a-d924-4d8f-a787-1e43cd9f6a5c-plugins-config-volume" pod "403f830a-d924-4d8f-a787-1e43cd9f6a5c" (UID: "403f830a-d924-4d8f-a787-1e43cd9f6a5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:10.219460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.219434 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/403f830a-d924-4d8f-a787-1e43cd9f6a5c-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:10.220551 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.220521 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403f830a-d924-4d8f-a787-1e43cd9f6a5c-kube-api-access-pfn8z" pod "403f830a-d924-4d8f-a787-1e43cd9f6a5c" (UID: "403f830a-d924-4d8f-a787-1e43cd9f6a5c"). InnerVolumeSpecName "kube-api-access-pfn8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:10.294926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.294821 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:10.299142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.299113 4828 generic.go:356] "Generic (PLEG): container finished" podID="403f830a-d924-4d8f-a787-1e43cd9f6a5c" containerID="0510ea6d346522ca30581245cf8cd6fc15d5f3fd72f176d7f61515e7bb700fe2" exitCode=0 Oct 30 17:22:10.299317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.299160 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" Oct 30 17:22:10.299317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.299198 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" event={"ID":"403f830a-d924-4d8f-a787-1e43cd9f6a5c","Type":"ContainerDied","Data":"0510ea6d346522ca30581245cf8cd6fc15d5f3fd72f176d7f61515e7bb700fe2"} Oct 30 17:22:10.299317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.299251 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt" event={"ID":"403f830a-d924-4d8f-a787-1e43cd9f6a5c","Type":"ContainerDied","Data":"556af197cc1608823a57ea86144aad184f03b2a2d15432921101eeba44e525b4"} Oct 30 17:22:10.299317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.299277 4828 scope.go:122] "RemoveContainer" containerID="0510ea6d346522ca30581245cf8cd6fc15d5f3fd72f176d7f61515e7bb700fe2" Oct 30 17:22:10.311893 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.311864 4828 scope.go:122] "RemoveContainer" containerID="0510ea6d346522ca30581245cf8cd6fc15d5f3fd72f176d7f61515e7bb700fe2" Oct 30 17:22:10.312276 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:22:10.312246 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0510ea6d346522ca30581245cf8cd6fc15d5f3fd72f176d7f61515e7bb700fe2\": container with ID starting with 0510ea6d346522ca30581245cf8cd6fc15d5f3fd72f176d7f61515e7bb700fe2 not found: ID does not exist" containerID="0510ea6d346522ca30581245cf8cd6fc15d5f3fd72f176d7f61515e7bb700fe2" Oct 30 17:22:10.312374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.312283 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0510ea6d346522ca30581245cf8cd6fc15d5f3fd72f176d7f61515e7bb700fe2"} err="failed to get container status \"0510ea6d346522ca30581245cf8cd6fc15d5f3fd72f176d7f61515e7bb700fe2\": rpc error: code = NotFound desc = could not find container \"0510ea6d346522ca30581245cf8cd6fc15d5f3fd72f176d7f61515e7bb700fe2\": container with ID starting with 0510ea6d346522ca30581245cf8cd6fc15d5f3fd72f176d7f61515e7bb700fe2 not found: ID does not exist" Oct 30 17:22:10.320834 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.320806 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfn8z\" (UniqueName: \"kubernetes.io/projected/403f830a-d924-4d8f-a787-1e43cd9f6a5c-kube-api-access-pfn8z\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:10.356948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.356863 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt"] Oct 30 17:22:10.362873 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.362845 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5d67ca-6dd886f747-lq2pt"] Oct 30 17:22:10.493645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:10.493609 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403f830a-d924-4d8f-a787-1e43cd9f6a5c" path="/var/lib/kubelet/pods/403f830a-d924-4d8f-a787-1e43cd9f6a5c/volumes" Oct 30 17:22:11.585877 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:11.585816 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:13.153090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:13.153051 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:13.636054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:13.636018 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:13.801846 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:13.801800 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:13.918756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:13.918639 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd"] Oct 30 17:22:13.919270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:13.918884 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" containerID="cri-o://c65d7a5c9f0e8c38a07aa56cddc0885cbcdedd6a2241c6864c8e48899409b5a4" gracePeriod=30 Oct 30 17:22:13.973680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:13.973643 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:15.103922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:15.099249 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:17.028192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:17.028157 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:18.578825 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:18.578788 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:19.255116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.255068 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:22:19.302621 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.302586 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istiod-ca-cert\") pod \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " Oct 30 17:22:19.303254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.303218 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-kube-api-access-c7cpr\" (UniqueName: \"kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-kube-api-access-c7cpr\") pod \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " Oct 30 17:22:19.303505 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.303341 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istiod-ca-cert" pod "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" (UID: "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:19.303661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.303643 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-socket\") pod \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " Oct 30 17:22:19.304142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.304124 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-credential-socket\") pod \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " Oct 30 17:22:19.304866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.304534 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-certs\") pod \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " Oct 30 17:22:19.304866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.304585 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-envoy\") pod \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " Oct 30 17:22:19.304866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.304825 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-token\" (UniqueName: \"kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-token\") pod \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " Oct 30 17:22:19.305432 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.304986 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-podinfo\") pod \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " Oct 30 17:22:19.305432 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.305046 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-data\") pod \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\" (UID: \"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4\") " Oct 30 17:22:19.305953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.303995 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-socket" pod "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" (UID: "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:19.305953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.304453 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-credential-socket" pod "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" (UID: "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:19.305953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.305817 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.305953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.305839 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.305953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.305853 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.305953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.305850 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-certs" pod "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" (UID: "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:19.306359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.306068 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-data" pod "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" (UID: "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:19.306984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.306963 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-kube-api-access-c7cpr" pod "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" (UID: "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4"). InnerVolumeSpecName "kube-api-access-c7cpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:19.308222 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.308192 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-envoy" pod "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" (UID: "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:19.308409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.308390 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-token" pod "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" (UID: "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:19.332580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.332512 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-podinfo" pod "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" (UID: "3d19b8e4-861b-4a5a-bb93-821c92e3a5e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:22:19.379510 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.379472 4828 generic.go:356] "Generic (PLEG): container finished" podID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerID="c65d7a5c9f0e8c38a07aa56cddc0885cbcdedd6a2241c6864c8e48899409b5a4" exitCode=0 Oct 30 17:22:19.379729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.379535 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" Oct 30 17:22:19.379729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.379588 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" event={"ID":"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4","Type":"ContainerDied","Data":"c65d7a5c9f0e8c38a07aa56cddc0885cbcdedd6a2241c6864c8e48899409b5a4"} Oct 30 17:22:19.379729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.379633 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd" event={"ID":"3d19b8e4-861b-4a5a-bb93-821c92e3a5e4","Type":"ContainerDied","Data":"5b2815f214e4be548180059411516fdbda65caf48613aa07a46af3fe32e9fe39"} Oct 30 17:22:19.379729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.379661 4828 scope.go:122] "RemoveContainer" containerID="c65d7a5c9f0e8c38a07aa56cddc0885cbcdedd6a2241c6864c8e48899409b5a4" Oct 30 17:22:19.396000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.395834 4828 scope.go:122] "RemoveContainer" containerID="3921be05aa428d4d55d5f0e1dbc05c2cbfea1131975e2a2d97e29f784c0b47cc" Oct 30 17:22:19.407597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.407571 4828 scope.go:122] "RemoveContainer" containerID="c65d7a5c9f0e8c38a07aa56cddc0885cbcdedd6a2241c6864c8e48899409b5a4" Oct 30 17:22:19.407823 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.407798 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.407917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.407830 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.407917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.407844 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.407917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.407858 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7cpr\" (UniqueName: \"kubernetes.io/projected/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-kube-api-access-c7cpr\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.407917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.407884 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.407917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.407894 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:19.408109 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:22:19.407929 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c65d7a5c9f0e8c38a07aa56cddc0885cbcdedd6a2241c6864c8e48899409b5a4\": container with ID starting with c65d7a5c9f0e8c38a07aa56cddc0885cbcdedd6a2241c6864c8e48899409b5a4 not found: ID does not exist" containerID="c65d7a5c9f0e8c38a07aa56cddc0885cbcdedd6a2241c6864c8e48899409b5a4" Oct 30 17:22:19.408109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.407970 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c65d7a5c9f0e8c38a07aa56cddc0885cbcdedd6a2241c6864c8e48899409b5a4"} err="failed to get container status \"c65d7a5c9f0e8c38a07aa56cddc0885cbcdedd6a2241c6864c8e48899409b5a4\": rpc error: code = NotFound desc = could not find container \"c65d7a5c9f0e8c38a07aa56cddc0885cbcdedd6a2241c6864c8e48899409b5a4\": container with ID starting with c65d7a5c9f0e8c38a07aa56cddc0885cbcdedd6a2241c6864c8e48899409b5a4 not found: ID does not exist" Oct 30 17:22:19.408109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.407995 4828 scope.go:122] "RemoveContainer" containerID="3921be05aa428d4d55d5f0e1dbc05c2cbfea1131975e2a2d97e29f784c0b47cc" Oct 30 17:22:19.408316 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:22:19.408295 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3921be05aa428d4d55d5f0e1dbc05c2cbfea1131975e2a2d97e29f784c0b47cc\": container with ID starting with 3921be05aa428d4d55d5f0e1dbc05c2cbfea1131975e2a2d97e29f784c0b47cc not found: ID does not exist" containerID="3921be05aa428d4d55d5f0e1dbc05c2cbfea1131975e2a2d97e29f784c0b47cc" Oct 30 17:22:19.408371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.408325 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3921be05aa428d4d55d5f0e1dbc05c2cbfea1131975e2a2d97e29f784c0b47cc"} err="failed to get container status \"3921be05aa428d4d55d5f0e1dbc05c2cbfea1131975e2a2d97e29f784c0b47cc\": rpc error: code = NotFound desc = could not find container \"3921be05aa428d4d55d5f0e1dbc05c2cbfea1131975e2a2d97e29f784c0b47cc\": container with ID starting with 3921be05aa428d4d55d5f0e1dbc05c2cbfea1131975e2a2d97e29f784c0b47cc not found: ID does not exist" Oct 30 17:22:19.437413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.437380 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd"] Oct 30 17:22:19.444251 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:19.444222 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d003a5-openshift-default-644f8657c-ptffd"] Oct 30 17:22:20.030361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:20.030323 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:20.662927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:20.662551 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" path="/var/lib/kubelet/pods/3d19b8e4-861b-4a5a-bb93-821c92e3a5e4/volumes" Oct 30 17:22:20.765173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:20.765134 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:20.914816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:20.914709 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:21.076642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:21.076605 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:21.442311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:21.442249 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:21.627928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:21.627097 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:22.904924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:22.903880 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:24.577921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:24.573870 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:24.776921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:24.776801 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:24.936383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:24.936294 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:26.312926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:26.296953 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:26.807103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:26.807066 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs"] Oct 30 17:22:26.861925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:26.853540 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:28.362722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:28.362683 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:28.528935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:28.528236 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:28.725208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:28.725046 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx"] Oct 30 17:22:28.725374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:28.725342 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" podUID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" containerName="istio-proxy" containerID="cri-o://b64a3bab6cb39621d145673320a19f87535b008627156f884f84b92b71e17cfb" gracePeriod=30 Oct 30 17:22:28.789054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:28.789020 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:30.071920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:30.068276 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:31.692931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:31.686874 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:33.826835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:33.826800 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:34.037431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.037288 4828 patch_prober.go:29] interesting pod/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 17:22:34.037431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.037359 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" podUID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 17:22:34.216681 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.216647 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:22:34.286540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.285196 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-envoy\") pod \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " Oct 30 17:22:34.286540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.285267 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-token\" (UniqueName: \"kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-token\") pod \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " Oct 30 17:22:34.286540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.285301 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-credential-socket\") pod \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " Oct 30 17:22:34.286540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.285376 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istiod-ca-cert\") pod \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " Oct 30 17:22:34.286540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.285446 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-podinfo\") pod \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " Oct 30 17:22:34.286540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.285473 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-certs\") pod \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " Oct 30 17:22:34.286540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.285576 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-data\") pod \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " Oct 30 17:22:34.286540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.285627 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-kube-api-access-pkwqt\" (UniqueName: \"kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-kube-api-access-pkwqt\") pod \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " Oct 30 17:22:34.286540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.285756 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-socket\") pod \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\" (UID: \"c1d4cafe-048c-46cf-98dc-381d6eff4d57\") " Oct 30 17:22:34.286540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.286186 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-data" pod "c1d4cafe-048c-46cf-98dc-381d6eff4d57" (UID: "c1d4cafe-048c-46cf-98dc-381d6eff4d57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:34.286540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.286231 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-certs" pod "c1d4cafe-048c-46cf-98dc-381d6eff4d57" (UID: "c1d4cafe-048c-46cf-98dc-381d6eff4d57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:34.286540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.286366 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-credential-socket" pod "c1d4cafe-048c-46cf-98dc-381d6eff4d57" (UID: "c1d4cafe-048c-46cf-98dc-381d6eff4d57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:34.286540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.286432 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istiod-ca-cert" pod "c1d4cafe-048c-46cf-98dc-381d6eff4d57" (UID: "c1d4cafe-048c-46cf-98dc-381d6eff4d57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:34.291307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.286718 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.291307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.286741 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.291307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.286753 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.291307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.286765 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.291307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.289697 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-kube-api-access-pkwqt" pod "c1d4cafe-048c-46cf-98dc-381d6eff4d57" (UID: "c1d4cafe-048c-46cf-98dc-381d6eff4d57"). InnerVolumeSpecName "kube-api-access-pkwqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:34.291307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.289985 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-podinfo" pod "c1d4cafe-048c-46cf-98dc-381d6eff4d57" (UID: "c1d4cafe-048c-46cf-98dc-381d6eff4d57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:22:34.291307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.290049 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-token" pod "c1d4cafe-048c-46cf-98dc-381d6eff4d57" (UID: "c1d4cafe-048c-46cf-98dc-381d6eff4d57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:34.291307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.290502 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-socket" pod "c1d4cafe-048c-46cf-98dc-381d6eff4d57" (UID: "c1d4cafe-048c-46cf-98dc-381d6eff4d57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:34.291794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.291413 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-envoy" pod "c1d4cafe-048c-46cf-98dc-381d6eff4d57" (UID: "c1d4cafe-048c-46cf-98dc-381d6eff4d57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:34.387950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.387896 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.387950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.387947 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkwqt\" (UniqueName: \"kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-kube-api-access-pkwqt\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.388163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.387963 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.388163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.387975 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.388163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.387987 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d4cafe-048c-46cf-98dc-381d6eff4d57-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:34.719593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.719509 4828 generic.go:356] "Generic (PLEG): container finished" podID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" containerID="b64a3bab6cb39621d145673320a19f87535b008627156f884f84b92b71e17cfb" exitCode=0 Oct 30 17:22:34.719593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.719556 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" Oct 30 17:22:34.719781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.719589 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" event={"ID":"c1d4cafe-048c-46cf-98dc-381d6eff4d57","Type":"ContainerDied","Data":"b64a3bab6cb39621d145673320a19f87535b008627156f884f84b92b71e17cfb"} Oct 30 17:22:34.719781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.719648 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx" event={"ID":"c1d4cafe-048c-46cf-98dc-381d6eff4d57","Type":"ContainerDied","Data":"4785c5dc1265cb6503630f6132fad3789604a2a3ae1979a05e0ed4c71021bba1"} Oct 30 17:22:34.719781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.719671 4828 scope.go:122] "RemoveContainer" containerID="b64a3bab6cb39621d145673320a19f87535b008627156f884f84b92b71e17cfb" Oct 30 17:22:34.730854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.730831 4828 scope.go:122] "RemoveContainer" containerID="b64a3bab6cb39621d145673320a19f87535b008627156f884f84b92b71e17cfb" Oct 30 17:22:34.731149 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:22:34.731129 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64a3bab6cb39621d145673320a19f87535b008627156f884f84b92b71e17cfb\": container with ID starting with b64a3bab6cb39621d145673320a19f87535b008627156f884f84b92b71e17cfb not found: ID does not exist" containerID="b64a3bab6cb39621d145673320a19f87535b008627156f884f84b92b71e17cfb" Oct 30 17:22:34.731212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.731159 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64a3bab6cb39621d145673320a19f87535b008627156f884f84b92b71e17cfb"} err="failed to get container status \"b64a3bab6cb39621d145673320a19f87535b008627156f884f84b92b71e17cfb\": rpc error: code = NotFound desc = could not find container \"b64a3bab6cb39621d145673320a19f87535b008627156f884f84b92b71e17cfb\": container with ID starting with b64a3bab6cb39621d145673320a19f87535b008627156f884f84b92b71e17cfb not found: ID does not exist" Oct 30 17:22:34.798984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.798911 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx"] Oct 30 17:22:34.815951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:34.815883 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5d67ca-openshift-default-589489f7d5-55lfx"] Oct 30 17:22:35.945924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:35.939447 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:36.495051 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:36.495019 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" path="/var/lib/kubelet/pods/c1d4cafe-048c-46cf-98dc-381d6eff4d57/volumes" Oct 30 17:22:37.812924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:37.812651 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:37.919949 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:22:37.919489 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:22:39.521594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:39.508767 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:41.047076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:41.047037 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:41.997372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:41.997324 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" podUID="92fad2ed-7234-450c-93cc-7c424235140d" containerName="main" containerID="cri-o://8118473804a2ba3603e94b95a5d03b43fc8f3b482e22c19709b0f84585b3dbf7" gracePeriod=45 Oct 30 17:22:42.321827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.321791 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:22:42.449312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.449280 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-home\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-home\") pod \"92fad2ed-7234-450c-93cc-7c424235140d\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " Oct 30 17:22:42.449527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.449348 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-tmp-dir\") pod \"92fad2ed-7234-450c-93cc-7c424235140d\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " Oct 30 17:22:42.449527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.449479 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-model-cache\") pod \"92fad2ed-7234-450c-93cc-7c424235140d\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " Oct 30 17:22:42.449657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.449589 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-dshm\") pod \"92fad2ed-7234-450c-93cc-7c424235140d\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " Oct 30 17:22:42.449717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.449652 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-tmp-dir" pod "92fad2ed-7234-450c-93cc-7c424235140d" (UID: "92fad2ed-7234-450c-93cc-7c424235140d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:42.449717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.449698 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-model-cache" pod "92fad2ed-7234-450c-93cc-7c424235140d" (UID: "92fad2ed-7234-450c-93cc-7c424235140d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:42.450054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.450031 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92fad2ed-7234-450c-93cc-7c424235140d-tls-certs\" (UniqueName: \"kubernetes.io/secret/92fad2ed-7234-450c-93cc-7c424235140d-tls-certs\") pod \"92fad2ed-7234-450c-93cc-7c424235140d\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " Oct 30 17:22:42.450320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.450303 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92fad2ed-7234-450c-93cc-7c424235140d-kube-api-access-qkvjc\" (UniqueName: \"kubernetes.io/projected/92fad2ed-7234-450c-93cc-7c424235140d-kube-api-access-qkvjc\") pod \"92fad2ed-7234-450c-93cc-7c424235140d\" (UID: \"92fad2ed-7234-450c-93cc-7c424235140d\") " Oct 30 17:22:42.451221 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.451201 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:42.451341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.451330 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:42.452168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.452149 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-home" pod "92fad2ed-7234-450c-93cc-7c424235140d" (UID: "92fad2ed-7234-450c-93cc-7c424235140d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:42.461493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.461457 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92fad2ed-7234-450c-93cc-7c424235140d-kube-api-access-qkvjc" pod "92fad2ed-7234-450c-93cc-7c424235140d" (UID: "92fad2ed-7234-450c-93cc-7c424235140d"). InnerVolumeSpecName "kube-api-access-qkvjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:42.463040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.463008 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92fad2ed-7234-450c-93cc-7c424235140d-tls-certs" pod "92fad2ed-7234-450c-93cc-7c424235140d" (UID: "92fad2ed-7234-450c-93cc-7c424235140d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:42.464682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.464657 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-dshm" pod "92fad2ed-7234-450c-93cc-7c424235140d" (UID: "92fad2ed-7234-450c-93cc-7c424235140d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:22:42.553250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.553161 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:42.553250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.553195 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fad2ed-7234-450c-93cc-7c424235140d-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:42.553250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.553205 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkvjc\" (UniqueName: \"kubernetes.io/projected/92fad2ed-7234-450c-93cc-7c424235140d-kube-api-access-qkvjc\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:42.553250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.553214 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fad2ed-7234-450c-93cc-7c424235140d-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:22:42.794060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.794022 4828 generic.go:356] "Generic (PLEG): container finished" podID="92fad2ed-7234-450c-93cc-7c424235140d" containerID="8118473804a2ba3603e94b95a5d03b43fc8f3b482e22c19709b0f84585b3dbf7" exitCode=0 Oct 30 17:22:42.794257 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.794083 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" Oct 30 17:22:42.794257 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.794102 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" event={"ID":"92fad2ed-7234-450c-93cc-7c424235140d","Type":"ContainerDied","Data":"8118473804a2ba3603e94b95a5d03b43fc8f3b482e22c19709b0f84585b3dbf7"} Oct 30 17:22:42.794257 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.794145 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs" event={"ID":"92fad2ed-7234-450c-93cc-7c424235140d","Type":"ContainerDied","Data":"87329eb93564d643b922d7b0b35fa8c58d814af110274123ab9dcb578a86a47a"} Oct 30 17:22:42.794257 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.794161 4828 scope.go:122] "RemoveContainer" containerID="8118473804a2ba3603e94b95a5d03b43fc8f3b482e22c19709b0f84585b3dbf7" Oct 30 17:22:42.806496 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.806471 4828 scope.go:122] "RemoveContainer" containerID="8118473804a2ba3603e94b95a5d03b43fc8f3b482e22c19709b0f84585b3dbf7" Oct 30 17:22:42.806861 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:22:42.806838 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8118473804a2ba3603e94b95a5d03b43fc8f3b482e22c19709b0f84585b3dbf7\": container with ID starting with 8118473804a2ba3603e94b95a5d03b43fc8f3b482e22c19709b0f84585b3dbf7 not found: ID does not exist" containerID="8118473804a2ba3603e94b95a5d03b43fc8f3b482e22c19709b0f84585b3dbf7" Oct 30 17:22:42.807026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.806873 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8118473804a2ba3603e94b95a5d03b43fc8f3b482e22c19709b0f84585b3dbf7"} err="failed to get container status \"8118473804a2ba3603e94b95a5d03b43fc8f3b482e22c19709b0f84585b3dbf7\": rpc error: code = NotFound desc = could not find container \"8118473804a2ba3603e94b95a5d03b43fc8f3b482e22c19709b0f84585b3dbf7\": container with ID starting with 8118473804a2ba3603e94b95a5d03b43fc8f3b482e22c19709b0f84585b3dbf7 not found: ID does not exist" Oct 30 17:22:42.850223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.850188 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs"] Oct 30 17:22:42.854850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.854817 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-82243b/auth-test-model-82243b-kserve-5cc4bdc969-672rs"] Oct 30 17:22:42.909025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:42.908991 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:44.495979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:44.495944 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92fad2ed-7234-450c-93cc-7c424235140d" path="/var/lib/kubelet/pods/92fad2ed-7234-450c-93cc-7c424235140d/volumes" Oct 30 17:22:44.571725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:44.571685 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:46.183923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:46.179976 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:47.693924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:47.693866 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:49.413303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:49.413262 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:51.046526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:51.046483 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:52.834434 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:52.834396 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:53.694998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:53.685688 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:54.497923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:54.494767 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:56.090284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:56.090244 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:56.630935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:56.629796 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:57.492776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:57.492736 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:58.113067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:58.113029 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:58.272190 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:58.272107 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:58.901773 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:58.901733 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:59.049303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:59.049268 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:59.207144 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:59.207051 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:22:59.561790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:22:59.561756 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:01.019968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:01.019931 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:02.561366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:02.561329 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:04.080479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:04.080439 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:04.242790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:04.242752 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:04.394102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:04.393993 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:06.044922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:06.044805 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:06.226985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:06.226940 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:06.384158 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:06.383992 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:06.506176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:06.506136 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8"] Oct 30 17:23:06.506456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:06.506430 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerName="istio-proxy" containerID="cri-o://4aa5f80314716a0898827446f8308b5146ccd8bb04b41aee5528cad7f5b90dc7" gracePeriod=30 Oct 30 17:23:06.586306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:06.586263 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:07.770549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:07.770514 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:07.914578 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:23:07.914541 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:23:08.652740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:08.652699 4828 patch_prober.go:29] interesting pod/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:23:08.653078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:08.653046 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:23:09.455879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:09.455821 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:11.063714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:11.063670 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:11.815501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:11.815467 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:23:12.010063 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.010021 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-data\") pod \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " Oct 30 17:23:12.010244 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.010074 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-certs\") pod \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " Oct 30 17:23:12.010244 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.010223 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-kube-api-access-s2z7v\" (UniqueName: \"kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-kube-api-access-s2z7v\") pod \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " Oct 30 17:23:12.010394 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.010258 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-credential-socket\") pod \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " Oct 30 17:23:12.010483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.010457 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-data" pod "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" (UID: "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:12.010549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.010472 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-certs" pod "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" (UID: "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:12.010549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.010521 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-socket\") pod \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " Oct 30 17:23:12.010667 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.010587 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-envoy\") pod \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " Oct 30 17:23:12.010667 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.010632 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-podinfo\") pod \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " Oct 30 17:23:12.010774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.010686 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istiod-ca-cert\") pod \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " Oct 30 17:23:12.010774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.010716 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-token\" (UniqueName: \"kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-token\") pod \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\" (UID: \"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0\") " Oct 30 17:23:12.011210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.010786 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-credential-socket" pod "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" (UID: "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:12.011210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.011167 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-socket" pod "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" (UID: "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:12.011686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.011665 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:12.012166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.012148 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:12.012310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.012296 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:12.012417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.012402 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:12.015208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.011785 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istiod-ca-cert" pod "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" (UID: "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:12.016535 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.015211 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-podinfo" pod "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" (UID: "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:23:12.016535 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.015683 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-kube-api-access-s2z7v" pod "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" (UID: "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0"). InnerVolumeSpecName "kube-api-access-s2z7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:12.016535 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.015826 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-token" pod "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" (UID: "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:12.016535 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.016435 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-envoy" pod "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" (UID: "330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:12.113484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.113445 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:12.113484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.113480 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:12.113484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.113490 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:12.113980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.113499 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:12.113980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.113508 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2z7v\" (UniqueName: \"kubernetes.io/projected/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0-kube-api-access-s2z7v\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:12.173112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.173069 4828 generic.go:356] "Generic (PLEG): container finished" podID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerID="4aa5f80314716a0898827446f8308b5146ccd8bb04b41aee5528cad7f5b90dc7" exitCode=0 Oct 30 17:23:12.173315 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.173129 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" Oct 30 17:23:12.173315 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.173149 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" event={"ID":"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0","Type":"ContainerDied","Data":"4aa5f80314716a0898827446f8308b5146ccd8bb04b41aee5528cad7f5b90dc7"} Oct 30 17:23:12.173315 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.173217 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8" event={"ID":"330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0","Type":"ContainerDied","Data":"e8181dbd09c8b7608dbc4fa05527db4c0009935ed745b55352f3bfee26f1ae88"} Oct 30 17:23:12.173315 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.173250 4828 scope.go:122] "RemoveContainer" containerID="4aa5f80314716a0898827446f8308b5146ccd8bb04b41aee5528cad7f5b90dc7" Oct 30 17:23:12.190750 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.190722 4828 scope.go:122] "RemoveContainer" containerID="daa4cf24bfd00b221e1daa64ff4584e2ff59118ee1b77d95d2572060caf4bac2" Oct 30 17:23:12.202361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.202335 4828 scope.go:122] "RemoveContainer" containerID="4aa5f80314716a0898827446f8308b5146ccd8bb04b41aee5528cad7f5b90dc7" Oct 30 17:23:12.202723 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:23:12.202692 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa5f80314716a0898827446f8308b5146ccd8bb04b41aee5528cad7f5b90dc7\": container with ID starting with 4aa5f80314716a0898827446f8308b5146ccd8bb04b41aee5528cad7f5b90dc7 not found: ID does not exist" containerID="4aa5f80314716a0898827446f8308b5146ccd8bb04b41aee5528cad7f5b90dc7" Oct 30 17:23:12.202806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.202736 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa5f80314716a0898827446f8308b5146ccd8bb04b41aee5528cad7f5b90dc7"} err="failed to get container status \"4aa5f80314716a0898827446f8308b5146ccd8bb04b41aee5528cad7f5b90dc7\": rpc error: code = NotFound desc = could not find container \"4aa5f80314716a0898827446f8308b5146ccd8bb04b41aee5528cad7f5b90dc7\": container with ID starting with 4aa5f80314716a0898827446f8308b5146ccd8bb04b41aee5528cad7f5b90dc7 not found: ID does not exist" Oct 30 17:23:12.202806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.202762 4828 scope.go:122] "RemoveContainer" containerID="daa4cf24bfd00b221e1daa64ff4584e2ff59118ee1b77d95d2572060caf4bac2" Oct 30 17:23:12.203061 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:23:12.203040 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa4cf24bfd00b221e1daa64ff4584e2ff59118ee1b77d95d2572060caf4bac2\": container with ID starting with daa4cf24bfd00b221e1daa64ff4584e2ff59118ee1b77d95d2572060caf4bac2 not found: ID does not exist" containerID="daa4cf24bfd00b221e1daa64ff4584e2ff59118ee1b77d95d2572060caf4bac2" Oct 30 17:23:12.203127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.203068 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa4cf24bfd00b221e1daa64ff4584e2ff59118ee1b77d95d2572060caf4bac2"} err="failed to get container status \"daa4cf24bfd00b221e1daa64ff4584e2ff59118ee1b77d95d2572060caf4bac2\": rpc error: code = NotFound desc = could not find container \"daa4cf24bfd00b221e1daa64ff4584e2ff59118ee1b77d95d2572060caf4bac2\": container with ID starting with daa4cf24bfd00b221e1daa64ff4584e2ff59118ee1b77d95d2572060caf4bac2 not found: ID does not exist" Oct 30 17:23:12.273313 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.273276 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8"] Oct 30 17:23:12.282410 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.282325 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c82666-openshift-default-767d987659-7j4f8"] Oct 30 17:23:12.496005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.495968 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" path="/var/lib/kubelet/pods/330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0/volumes" Oct 30 17:23:12.709979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:12.704710 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:13.191634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:13.191593 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:14.340116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:14.340064 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:16.062525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:16.062482 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:17.651921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:17.649054 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:19.347929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:19.345390 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:20.918195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:20.918162 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:22.433228 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:22.433176 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:23.979923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:23.978255 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:25.626927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:25.626520 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:27.001974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:27.001923 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:27.712664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:27.712626 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:27.933920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:27.933213 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:28.126544 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:28.126506 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:28.463411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:28.463316 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:29.640925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:29.608455 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:29.895175 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:29.895081 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22"] Oct 30 17:23:30.502081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:30.502027 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:23:31.030229 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:31.015478 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:31.179460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:31.179418 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:31.265921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:31.263681 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:23:32.519618 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:32.519580 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:34.302463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:34.302428 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:34.938958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:34.938916 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:35.091519 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:35.091483 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:35.266719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:35.266681 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt"] Oct 30 17:23:35.267245 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:35.267209 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" containerID="cri-o://c21fd047f78720edd77463dee0fa38b866b64bff3e15487a8b18b814982c53c5" gracePeriod=30 Oct 30 17:23:35.320985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:35.320924 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q"] Oct 30 17:23:35.321515 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:35.321173 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerName="istio-proxy" containerID="cri-o://dab5bf9b8df1c23d2ec873bd2f621ed18bf94e3b4676c549bea1808dcb4f2057" gracePeriod=30 Oct 30 17:23:35.369005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:35.368583 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:36.596122 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:36.596086 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:37.923674 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:23:37.923641 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:23:38.358380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:38.358277 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:39.846824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:39.846782 4828 patch_prober.go:29] interesting pod/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:23:39.847224 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:39.846863 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:23:40.074954 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.066622 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:40.560304 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.556383 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:23:40.703357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.703045 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.738769 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-certs\") pod \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.738828 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istiod-ca-cert\") pod \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.738882 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-data\") pod \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.738992 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-kube-api-access-kgckb\" (UniqueName: \"kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-kube-api-access-kgckb\") pod \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.739141 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-socket\") pod \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.739193 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-envoy\") pod \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.739135 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-certs" pod "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" (UID: "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.739321 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-data" pod "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" (UID: "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.739340 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-token\" (UniqueName: \"kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-token\") pod \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.739431 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-socket" pod "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" (UID: "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.739492 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-podinfo\") pod \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.739540 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-credential-socket\") pod \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\" (UID: \"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a\") " Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.739874 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-credential-socket" pod "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" (UID: "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.740017 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istiod-ca-cert" pod "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" (UID: "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.740268 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.740283 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.740293 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.740301 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.740360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.740312 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.742270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.742229 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-token" pod "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" (UID: "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:40.742466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.742438 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-podinfo" pod "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" (UID: "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:23:40.742545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.742486 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-envoy" pod "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" (UID: "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.742634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.742609 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-kube-api-access-kgckb" pod "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" (UID: "9a7d85b1-1928-43bb-9f10-6a76f1d88a2a"). InnerVolumeSpecName "kube-api-access-kgckb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:40.840872 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.840776 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-podinfo\") pod \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " Oct 30 17:23:40.840872 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.840841 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-envoy\") pod \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " Oct 30 17:23:40.841179 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.841019 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-token\" (UniqueName: \"kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-token\") pod \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " Oct 30 17:23:40.841179 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.841081 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-socket\") pod \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " Oct 30 17:23:40.841179 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.841137 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-credential-socket\") pod \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " Oct 30 17:23:40.841364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.841246 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-data\") pod \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " Oct 30 17:23:40.841364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.841325 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istiod-ca-cert\") pod \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " Oct 30 17:23:40.841481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.841380 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-socket" pod "4e26d484-11cd-4ef4-a019-6fe388fa0e9d" (UID: "4e26d484-11cd-4ef4-a019-6fe388fa0e9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.841481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.841401 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-certs\") pod \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " Oct 30 17:23:40.841596 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.841474 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-credential-socket" pod "4e26d484-11cd-4ef4-a019-6fe388fa0e9d" (UID: "4e26d484-11cd-4ef4-a019-6fe388fa0e9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.841596 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.841514 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-data" pod "4e26d484-11cd-4ef4-a019-6fe388fa0e9d" (UID: "4e26d484-11cd-4ef4-a019-6fe388fa0e9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.841711 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.841590 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-kube-api-access-bx7lp\" (UniqueName: \"kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-kube-api-access-bx7lp\") pod \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\" (UID: \"4e26d484-11cd-4ef4-a019-6fe388fa0e9d\") " Oct 30 17:23:40.841711 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.841635 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-certs" pod "4e26d484-11cd-4ef4-a019-6fe388fa0e9d" (UID: "4e26d484-11cd-4ef4-a019-6fe388fa0e9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.841812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.841767 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istiod-ca-cert" pod "4e26d484-11cd-4ef4-a019-6fe388fa0e9d" (UID: "4e26d484-11cd-4ef4-a019-6fe388fa0e9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:40.842478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.842443 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.842478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.842477 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.842655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.842492 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.842655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.842504 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.842655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.842518 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.842655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.842531 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgckb\" (UniqueName: \"kubernetes.io/projected/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-kube-api-access-kgckb\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.842655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.842545 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.842655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.842557 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.842655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.842569 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.843540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.843509 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-token" pod "4e26d484-11cd-4ef4-a019-6fe388fa0e9d" (UID: "4e26d484-11cd-4ef4-a019-6fe388fa0e9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:40.843652 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.843634 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-podinfo" pod "4e26d484-11cd-4ef4-a019-6fe388fa0e9d" (UID: "4e26d484-11cd-4ef4-a019-6fe388fa0e9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:23:40.843864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.843836 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-envoy" pod "4e26d484-11cd-4ef4-a019-6fe388fa0e9d" (UID: "4e26d484-11cd-4ef4-a019-6fe388fa0e9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:40.844000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.843980 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-kube-api-access-bx7lp" pod "4e26d484-11cd-4ef4-a019-6fe388fa0e9d" (UID: "4e26d484-11cd-4ef4-a019-6fe388fa0e9d"). InnerVolumeSpecName "kube-api-access-bx7lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:40.943998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.943955 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx7lp\" (UniqueName: \"kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-kube-api-access-bx7lp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.943998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.943993 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.943998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.944007 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:40.944469 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:40.944019 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e26d484-11cd-4ef4-a019-6fe388fa0e9d-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.193405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.193372 4828 generic.go:356] "Generic (PLEG): container finished" podID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerID="dab5bf9b8df1c23d2ec873bd2f621ed18bf94e3b4676c549bea1808dcb4f2057" exitCode=0 Oct 30 17:23:41.193646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.193585 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" event={"ID":"4e26d484-11cd-4ef4-a019-6fe388fa0e9d","Type":"ContainerDied","Data":"dab5bf9b8df1c23d2ec873bd2f621ed18bf94e3b4676c549bea1808dcb4f2057"} Oct 30 17:23:41.193646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.193643 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" event={"ID":"4e26d484-11cd-4ef4-a019-6fe388fa0e9d","Type":"ContainerDied","Data":"d7bfe4e7947423ae02f966e1ab0b524cc91b48b40c22e51ae16c7098cb0a32e6"} Oct 30 17:23:41.193771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.193672 4828 scope.go:122] "RemoveContainer" containerID="dab5bf9b8df1c23d2ec873bd2f621ed18bf94e3b4676c549bea1808dcb4f2057" Oct 30 17:23:41.193953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.193852 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q" Oct 30 17:23:41.199988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.199957 4828 generic.go:356] "Generic (PLEG): container finished" podID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerID="c21fd047f78720edd77463dee0fa38b866b64bff3e15487a8b18b814982c53c5" exitCode=0 Oct 30 17:23:41.200103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.200030 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" Oct 30 17:23:41.200177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.200132 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" event={"ID":"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a","Type":"ContainerDied","Data":"c21fd047f78720edd77463dee0fa38b866b64bff3e15487a8b18b814982c53c5"} Oct 30 17:23:41.200177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.200172 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt" event={"ID":"9a7d85b1-1928-43bb-9f10-6a76f1d88a2a","Type":"ContainerDied","Data":"0876774a7b33350c0f2beadd7464f5f6495b91674d941db68e0c1f5ec6e3b016"} Oct 30 17:23:41.230609 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.230583 4828 scope.go:122] "RemoveContainer" containerID="dab5bf9b8df1c23d2ec873bd2f621ed18bf94e3b4676c549bea1808dcb4f2057" Oct 30 17:23:41.230881 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:23:41.230863 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab5bf9b8df1c23d2ec873bd2f621ed18bf94e3b4676c549bea1808dcb4f2057\": container with ID starting with dab5bf9b8df1c23d2ec873bd2f621ed18bf94e3b4676c549bea1808dcb4f2057 not found: ID does not exist" containerID="dab5bf9b8df1c23d2ec873bd2f621ed18bf94e3b4676c549bea1808dcb4f2057" Oct 30 17:23:41.230992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.230888 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab5bf9b8df1c23d2ec873bd2f621ed18bf94e3b4676c549bea1808dcb4f2057"} err="failed to get container status \"dab5bf9b8df1c23d2ec873bd2f621ed18bf94e3b4676c549bea1808dcb4f2057\": rpc error: code = NotFound desc = could not find container \"dab5bf9b8df1c23d2ec873bd2f621ed18bf94e3b4676c549bea1808dcb4f2057\": container with ID starting with dab5bf9b8df1c23d2ec873bd2f621ed18bf94e3b4676c549bea1808dcb4f2057 not found: ID does not exist" Oct 30 17:23:41.230992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.230936 4828 scope.go:122] "RemoveContainer" containerID="c21fd047f78720edd77463dee0fa38b866b64bff3e15487a8b18b814982c53c5" Oct 30 17:23:41.242137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.242114 4828 scope.go:122] "RemoveContainer" containerID="9ff64fd8dd9db293aa8e03fbf2fc3ddc9b3204f7350a532bf9f1bd50a20c04d3" Oct 30 17:23:41.253133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.253108 4828 scope.go:122] "RemoveContainer" containerID="c21fd047f78720edd77463dee0fa38b866b64bff3e15487a8b18b814982c53c5" Oct 30 17:23:41.253432 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:23:41.253403 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21fd047f78720edd77463dee0fa38b866b64bff3e15487a8b18b814982c53c5\": container with ID starting with c21fd047f78720edd77463dee0fa38b866b64bff3e15487a8b18b814982c53c5 not found: ID does not exist" containerID="c21fd047f78720edd77463dee0fa38b866b64bff3e15487a8b18b814982c53c5" Oct 30 17:23:41.253525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.253439 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21fd047f78720edd77463dee0fa38b866b64bff3e15487a8b18b814982c53c5"} err="failed to get container status \"c21fd047f78720edd77463dee0fa38b866b64bff3e15487a8b18b814982c53c5\": rpc error: code = NotFound desc = could not find container \"c21fd047f78720edd77463dee0fa38b866b64bff3e15487a8b18b814982c53c5\": container with ID starting with c21fd047f78720edd77463dee0fa38b866b64bff3e15487a8b18b814982c53c5 not found: ID does not exist" Oct 30 17:23:41.253525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.253457 4828 scope.go:122] "RemoveContainer" containerID="9ff64fd8dd9db293aa8e03fbf2fc3ddc9b3204f7350a532bf9f1bd50a20c04d3" Oct 30 17:23:41.253739 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:23:41.253714 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff64fd8dd9db293aa8e03fbf2fc3ddc9b3204f7350a532bf9f1bd50a20c04d3\": container with ID starting with 9ff64fd8dd9db293aa8e03fbf2fc3ddc9b3204f7350a532bf9f1bd50a20c04d3 not found: ID does not exist" containerID="9ff64fd8dd9db293aa8e03fbf2fc3ddc9b3204f7350a532bf9f1bd50a20c04d3" Oct 30 17:23:41.253795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.253766 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff64fd8dd9db293aa8e03fbf2fc3ddc9b3204f7350a532bf9f1bd50a20c04d3"} err="failed to get container status \"9ff64fd8dd9db293aa8e03fbf2fc3ddc9b3204f7350a532bf9f1bd50a20c04d3\": rpc error: code = NotFound desc = could not find container \"9ff64fd8dd9db293aa8e03fbf2fc3ddc9b3204f7350a532bf9f1bd50a20c04d3\": container with ID starting with 9ff64fd8dd9db293aa8e03fbf2fc3ddc9b3204f7350a532bf9f1bd50a20c04d3 not found: ID does not exist" Oct 30 17:23:41.270743 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.270703 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt"] Oct 30 17:23:41.281196 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.281159 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-bvdmt"] Oct 30 17:23:41.311087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.311016 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q"] Oct 30 17:23:41.316711 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.316682 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7c52f8-openshift-default-788cfdb5b5-j4c9q"] Oct 30 17:23:41.584201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.584166 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 17:23:41.584408 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.584235 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 17:23:41.738984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:41.738945 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:42.212842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:42.212811 4828 generic.go:356] "Generic (PLEG): container finished" podID="383c4614-f032-4952-9be1-d69075b86db8" containerID="81e498bc4f22ef97f4e982dec3ad06d920c8ac1f8f8728ab29e2e5cf6a701c8c" exitCode=0 Oct 30 17:23:42.213316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:42.212890 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" event={"ID":"383c4614-f032-4952-9be1-d69075b86db8","Type":"ContainerDied","Data":"81e498bc4f22ef97f4e982dec3ad06d920c8ac1f8f8728ab29e2e5cf6a701c8c"} Oct 30 17:23:42.213493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:42.213474 4828 scope.go:122] "RemoveContainer" containerID="81e498bc4f22ef97f4e982dec3ad06d920c8ac1f8f8728ab29e2e5cf6a701c8c" Oct 30 17:23:42.215957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:42.215883 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:23:42.216073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:42.216017 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:23:42.216073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:42.216066 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:23:42.497315 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:42.497222 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" path="/var/lib/kubelet/pods/4e26d484-11cd-4ef4-a019-6fe388fa0e9d/volumes" Oct 30 17:23:42.498088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:42.498065 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" path="/var/lib/kubelet/pods/9a7d85b1-1928-43bb-9f10-6a76f1d88a2a/volumes" Oct 30 17:23:43.225267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:43.225227 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" event={"ID":"383c4614-f032-4952-9be1-d69075b86db8","Type":"ContainerStarted","Data":"c61a207b0930f0a675be70d947eeaa1d874239d4cae15cd9e4fac5a961c376c4"} Oct 30 17:23:43.663934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:43.659664 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:43.796459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:43.796408 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:23:43.796833 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:43.796798 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 17:23:43.797018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:43.796985 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 17:23:44.799980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:44.787318 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 17:23:44.799980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:44.787401 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 17:23:45.219784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.219665 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" podUID="afe970bc-4f76-41e9-93c1-f18b36259ae3" containerName="main" containerID="cri-o://d9eebbeafd0c1c0fed3842d6b8f5e3e982994a096533997e091a4bff35f1d9a0" gracePeriod=45 Oct 30 17:23:45.238530 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.238491 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" podUID="afe970bc-4f76-41e9-93c1-f18b36259ae3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.176:8000/ready\": dial tcp 10.132.0.176:8000: connect: connection refused" Oct 30 17:23:45.404923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.379783 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:45.748840 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.748808 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:23:45.800922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.794762 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 17:23:45.800922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.794823 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 17:23:45.852714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.852607 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe970bc-4f76-41e9-93c1-f18b36259ae3-tls-certs\" (UniqueName: \"kubernetes.io/secret/afe970bc-4f76-41e9-93c1-f18b36259ae3-tls-certs\") pod \"afe970bc-4f76-41e9-93c1-f18b36259ae3\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " Oct 30 17:23:45.852714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.852663 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-tmp-dir\") pod \"afe970bc-4f76-41e9-93c1-f18b36259ae3\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " Oct 30 17:23:45.852993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.852852 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-home\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-home\") pod \"afe970bc-4f76-41e9-93c1-f18b36259ae3\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " Oct 30 17:23:45.852993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.852917 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-model-cache\") pod \"afe970bc-4f76-41e9-93c1-f18b36259ae3\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " Oct 30 17:23:45.853119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.852997 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe970bc-4f76-41e9-93c1-f18b36259ae3-kube-api-access-lp56z\" (UniqueName: \"kubernetes.io/projected/afe970bc-4f76-41e9-93c1-f18b36259ae3-kube-api-access-lp56z\") pod \"afe970bc-4f76-41e9-93c1-f18b36259ae3\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " Oct 30 17:23:45.853119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.853024 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-dshm\") pod \"afe970bc-4f76-41e9-93c1-f18b36259ae3\" (UID: \"afe970bc-4f76-41e9-93c1-f18b36259ae3\") " Oct 30 17:23:45.853119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.853073 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-tmp-dir" pod "afe970bc-4f76-41e9-93c1-f18b36259ae3" (UID: "afe970bc-4f76-41e9-93c1-f18b36259ae3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:45.853313 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.853264 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-model-cache" pod "afe970bc-4f76-41e9-93c1-f18b36259ae3" (UID: "afe970bc-4f76-41e9-93c1-f18b36259ae3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:45.853383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.853324 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-home" pod "afe970bc-4f76-41e9-93c1-f18b36259ae3" (UID: "afe970bc-4f76-41e9-93c1-f18b36259ae3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:45.853776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.853753 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:45.853776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.853776 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:45.854020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.853789 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:45.855987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.855870 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-dshm" pod "afe970bc-4f76-41e9-93c1-f18b36259ae3" (UID: "afe970bc-4f76-41e9-93c1-f18b36259ae3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:45.855987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.855942 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe970bc-4f76-41e9-93c1-f18b36259ae3-tls-certs" pod "afe970bc-4f76-41e9-93c1-f18b36259ae3" (UID: "afe970bc-4f76-41e9-93c1-f18b36259ae3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:45.857118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.857091 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe970bc-4f76-41e9-93c1-f18b36259ae3-kube-api-access-lp56z" pod "afe970bc-4f76-41e9-93c1-f18b36259ae3" (UID: "afe970bc-4f76-41e9-93c1-f18b36259ae3"). InnerVolumeSpecName "kube-api-access-lp56z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:45.955366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.955324 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp56z\" (UniqueName: \"kubernetes.io/projected/afe970bc-4f76-41e9-93c1-f18b36259ae3-kube-api-access-lp56z\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:45.955366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.955364 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afe970bc-4f76-41e9-93c1-f18b36259ae3-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:45.955366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:45.955374 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afe970bc-4f76-41e9-93c1-f18b36259ae3-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:23:46.208694 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.208454 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:46.273926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.271376 4828 generic.go:356] "Generic (PLEG): container finished" podID="afe970bc-4f76-41e9-93c1-f18b36259ae3" containerID="d9eebbeafd0c1c0fed3842d6b8f5e3e982994a096533997e091a4bff35f1d9a0" exitCode=0 Oct 30 17:23:46.273926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.271642 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" event={"ID":"afe970bc-4f76-41e9-93c1-f18b36259ae3","Type":"ContainerDied","Data":"d9eebbeafd0c1c0fed3842d6b8f5e3e982994a096533997e091a4bff35f1d9a0"} Oct 30 17:23:46.273926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.271683 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" event={"ID":"afe970bc-4f76-41e9-93c1-f18b36259ae3","Type":"ContainerDied","Data":"6c227b91b00c5e12fe13f0049dbc11c5b18aa017d7737d3ee4c5df8bebf72d04"} Oct 30 17:23:46.273926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.271704 4828 scope.go:122] "RemoveContainer" containerID="d9eebbeafd0c1c0fed3842d6b8f5e3e982994a096533997e091a4bff35f1d9a0" Oct 30 17:23:46.273926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.271918 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22" Oct 30 17:23:46.362935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.362435 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:46.414219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.414155 4828 scope.go:122] "RemoveContainer" containerID="d9eebbeafd0c1c0fed3842d6b8f5e3e982994a096533997e091a4bff35f1d9a0" Oct 30 17:23:46.414846 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:23:46.414804 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9eebbeafd0c1c0fed3842d6b8f5e3e982994a096533997e091a4bff35f1d9a0\": container with ID starting with d9eebbeafd0c1c0fed3842d6b8f5e3e982994a096533997e091a4bff35f1d9a0 not found: ID does not exist" containerID="d9eebbeafd0c1c0fed3842d6b8f5e3e982994a096533997e091a4bff35f1d9a0" Oct 30 17:23:46.415035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.414845 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9eebbeafd0c1c0fed3842d6b8f5e3e982994a096533997e091a4bff35f1d9a0"} err="failed to get container status \"d9eebbeafd0c1c0fed3842d6b8f5e3e982994a096533997e091a4bff35f1d9a0\": rpc error: code = NotFound desc = could not find container \"d9eebbeafd0c1c0fed3842d6b8f5e3e982994a096533997e091a4bff35f1d9a0\": container with ID starting with d9eebbeafd0c1c0fed3842d6b8f5e3e982994a096533997e091a4bff35f1d9a0 not found: ID does not exist" Oct 30 17:23:46.421438 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.421398 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22"] Oct 30 17:23:46.428452 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.428422 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2d01ec/test-model-2d01ec-kserve-9b455ff7-hxq22"] Oct 30 17:23:46.522837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.521511 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe970bc-4f76-41e9-93c1-f18b36259ae3" path="/var/lib/kubelet/pods/afe970bc-4f76-41e9-93c1-f18b36259ae3/volumes" Oct 30 17:23:46.791983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.791394 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 17:23:46.791983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:46.791475 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 17:23:47.283958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:47.278447 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:47.786391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:47.786350 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 17:23:47.786739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:47.786702 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 17:23:48.786383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:48.786331 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 17:23:48.786975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:48.786413 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 17:23:49.135119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:49.135035 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:49.792575 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:49.792536 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:23:49.793109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:49.793084 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:23:49.794012 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:49.793993 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:23:50.808212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:50.805499 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:51.767027 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:51.766990 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:52.494874 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:52.493413 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:54.266602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:54.266563 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:55.949189 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:55.949147 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:57.693880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:57.693842 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:58.339602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:58.339566 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:23:59.590106 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:23:59.590065 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:00.980853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.980814 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls"] Oct 30 17:24:00.982017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.981996 4828 cpu_manager.go:414] "containerMap: removing container" podUID="92fad2ed-7234-450c-93cc-7c424235140d" containerName="main" Oct 30 17:24:00.982096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982020 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92fad2ed-7234-450c-93cc-7c424235140d" containerName="main" Oct 30 17:24:00.982096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982047 4828 cpu_manager.go:414] "containerMap: removing container" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" Oct 30 17:24:00.982096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982055 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" Oct 30 17:24:00.982096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982069 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" containerName="istio-proxy" Oct 30 17:24:00.982096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982079 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" containerName="istio-proxy" Oct 30 17:24:00.982096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982094 4828 cpu_manager.go:414] "containerMap: removing container" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982103 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982117 4828 cpu_manager.go:414] "containerMap: removing container" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982124 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982137 4828 cpu_manager.go:414] "containerMap: removing container" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerName="istio-proxy" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982144 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerName="istio-proxy" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982166 4828 cpu_manager.go:414] "containerMap: removing container" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982174 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982198 4828 cpu_manager.go:414] "containerMap: removing container" podUID="403f830a-d924-4d8f-a787-1e43cd9f6a5c" containerName="payload-pre-processing" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982206 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403f830a-d924-4d8f-a787-1e43cd9f6a5c" containerName="payload-pre-processing" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982217 4828 cpu_manager.go:414] "containerMap: removing container" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerName="istio-proxy" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982224 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerName="istio-proxy" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982245 4828 cpu_manager.go:414] "containerMap: removing container" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerName="istio-proxy" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982251 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerName="istio-proxy" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982279 4828 cpu_manager.go:414] "containerMap: removing container" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" Oct 30 17:24:00.982288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982287 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300e1607-7771-4378-b40a-56cf985f25d5" containerName="istio-proxy" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982297 4828 cpu_manager.go:414] "containerMap: removing container" podUID="afe970bc-4f76-41e9-93c1-f18b36259ae3" containerName="main" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982304 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe970bc-4f76-41e9-93c1-f18b36259ae3" containerName="main" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982331 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c6cb4e0e-750c-4846-ab7c-375ac9fbe956" containerName="maas-api" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982337 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6cb4e0e-750c-4846-ab7c-375ac9fbe956" containerName="maas-api" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982504 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe970bc-4f76-41e9-93c1-f18b36259ae3" containerName="main" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982521 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6cb4e0e-750c-4846-ab7c-375ac9fbe956" containerName="maas-api" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982533 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="92fad2ed-7234-450c-93cc-7c424235140d" containerName="main" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982543 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982556 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d4cafe-048c-46cf-98dc-381d6eff4d57" containerName="istio-proxy" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982565 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7d85b1-1928-43bb-9f10-6a76f1d88a2a" containerName="istio-proxy" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982573 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982582 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerName="istio-proxy" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982593 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d19b8e4-861b-4a5a-bb93-821c92e3a5e4" containerName="istio-proxy" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982605 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="403f830a-d924-4d8f-a787-1e43cd9f6a5c" containerName="payload-pre-processing" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982615 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e26d484-11cd-4ef4-a019-6fe388fa0e9d" containerName="istio-proxy" Oct 30 17:24:00.982746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.982626 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="330fe0a1-61b1-42d0-a2d9-ca7e9b3ce1d0" containerName="istio-proxy" Oct 30 17:24:00.987234 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:00.987210 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.017859 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.017822 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-987dfc-openshift-default-dockercfg-ggt87\"" Oct 30 17:24:01.025610 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.025572 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls"] Oct 30 17:24:01.044007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.043966 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-certs\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.044212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.044021 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-podinfo\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.044308 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.044277 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istiod-ca-cert\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.044372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.044331 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-token\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.044520 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.044494 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx9k6\" (UniqueName: \"kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-kube-api-access-qx9k6\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.044613 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.044591 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.044671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.044644 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-envoy\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.044734 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.044712 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-credential-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.044792 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.044745 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-data\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.146090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.146050 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-data\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.146270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.146133 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-certs\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.146270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.146151 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-podinfo\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.146416 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.146276 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istiod-ca-cert\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.146416 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.146317 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-token\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.146416 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.146406 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx9k6\" (UniqueName: \"kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-kube-api-access-qx9k6\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.146582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.146457 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.146857 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.146548 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-data\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.146964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.146573 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-certs\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.146964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.146746 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.146964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.146829 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-envoy\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.146964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.146948 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-credential-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.147251 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.147226 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-credential-socket\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.147346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.147320 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istiod-ca-cert\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.148885 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.148860 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-envoy\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.149056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.149036 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-podinfo\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.171167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.171132 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-token\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.171167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.171148 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx9k6\" (UniqueName: \"kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-kube-api-access-qx9k6\") pod \"e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.230844 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.230403 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:01.302509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.302466 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:01.633433 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:24:01.633398 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49de0e3_8b4f_4f8a_b14c_aa94892f6dee.slice/crio-027b9090afe866bd27977540175d3a05db55d514e8617f70e4c2b3b5bdba4d27 WatchSource:0}: Error finding container 027b9090afe866bd27977540175d3a05db55d514e8617f70e4c2b3b5bdba4d27: Status 404 returned error can't find the container with id 027b9090afe866bd27977540175d3a05db55d514e8617f70e4c2b3b5bdba4d27 Oct 30 17:24:01.634479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.634449 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls"] Oct 30 17:24:01.636364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.636333 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:01.636438 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.636394 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:01.636438 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:01.636434 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:02.451710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:02.451662 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" event={"ID":"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee","Type":"ContainerStarted","Data":"bb865f51b48a7ef762794a788c6775c134649a3e260a7b56d96c5617d979660b"} Oct 30 17:24:02.452237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:02.451765 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" event={"ID":"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee","Type":"ContainerStarted","Data":"027b9090afe866bd27977540175d3a05db55d514e8617f70e4c2b3b5bdba4d27"} Oct 30 17:24:02.513099 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:02.513049 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" podStartSLOduration=2.513034229 podStartE2EDuration="2.513034229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:02.512418724 +0000 UTC m=+2220.157928908" watchObservedRunningTime="2026-10-30 17:24:02.513034229 +0000 UTC m=+2220.158544409" Oct 30 17:24:02.962278 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:02.962230 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:03.309601 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:03.309564 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:03.316057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:03.316021 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 17:24:03.317555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:03.317522 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" podUID="e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 17:24:03.825565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:03.825516 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:04.303493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:04.303457 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 17:24:04.306397 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:04.306361 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" podUID="e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 17:24:05.044977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:05.044147 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:05.306208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:05.306114 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 17:24:05.306528 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:05.306486 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" podUID="e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 17:24:06.303439 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:06.303397 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 17:24:06.304092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:06.304053 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" podUID="e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 17:24:06.768365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:06.768327 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:07.304332 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:07.304291 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 17:24:07.305133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:07.305078 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" podUID="e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 17:24:07.720670 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:07.720567 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:08.010024 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:24:08.009986 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:24:08.442498 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:08.442412 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:08.443418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:08.443394 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:08.444444 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:08.444416 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:24:08.505772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:08.505663 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:10.242754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:10.242706 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:11.563169 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:11.563070 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km"] Oct 30 17:24:11.619565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:11.619522 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:12.330255 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:12.330217 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:13.210578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:13.210539 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:14.431482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:14.431446 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:16.163752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:16.163706 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:17.877519 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:17.877460 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:19.344414 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:19.344383 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:20.023663 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:20.023607 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:20.219349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:20.219295 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:20.393782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:20.393678 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:20.765269 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:20.765227 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:20.992811 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:20.992766 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p"] Oct 30 17:24:20.993093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:20.993039 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" containerID="cri-o://a039b243aa94e9460182d4eeddd0a396c1c12a3f6ad8ac66b8698fb35bf639df" gracePeriod=30 Oct 30 17:24:21.010497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:21.010460 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8"] Oct 30 17:24:21.010763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:21.010693 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" containerID="cri-o://fa3d01ee6f657022887cfd7e37e34388db433fb66a84cf043d89e24ebcf40a45" gracePeriod=30 Oct 30 17:24:21.487176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:21.487141 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb"] Oct 30 17:24:21.631612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:21.631577 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:21.988930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:21.970741 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:22.761925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:22.761556 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:24.383256 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:24.383099 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:25.790738 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:25.790697 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn"] Oct 30 17:24:25.797230 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:25.797204 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:25.826677 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:25.826633 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn"] Oct 30 17:24:25.913391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:25.913347 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab756a2-144f-4411-85db-dcb56009529f-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:25.913660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:25.913407 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab756a2-144f-4411-85db-dcb56009529f-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:25.913660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:25.913582 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:25.913818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:25.913697 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:25.913818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:25.913726 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:25.913818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:25.913792 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgsbw\" (UniqueName: \"kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-kube-api-access-kgsbw\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:25.914038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:25.913889 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:25.914038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:25.914009 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:25.914152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:25.914053 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.018097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.017074 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.018097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.017208 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.018097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.017241 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.018097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.017276 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgsbw\" (UniqueName: \"kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-kube-api-access-kgsbw\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.018097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.017392 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.018097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.017417 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.018097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.017464 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.018097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.017644 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab756a2-144f-4411-85db-dcb56009529f-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.018097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.017684 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab756a2-144f-4411-85db-dcb56009529f-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.020163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.018698 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab756a2-144f-4411-85db-dcb56009529f-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.020163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.019041 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.020163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.019348 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.020163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.019618 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.052087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.046824 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.068525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.068488 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:26.071665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.070828 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.076291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.074228 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab756a2-144f-4411-85db-dcb56009529f-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.076706 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.076685 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgsbw\" (UniqueName: \"kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-kube-api-access-kgsbw\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.087472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.087448 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.113548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.113522 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:26.330348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.330250 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:24:26.386343 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.386315 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:24:26.425097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.425065 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-socket\") pod \"a04e6e28-2061-4325-ae70-980c86ae72e8\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " Oct 30 17:24:26.425289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.425120 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-data\") pod \"a04e6e28-2061-4325-ae70-980c86ae72e8\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " Oct 30 17:24:26.425289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.425146 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-credential-socket\") pod \"a04e6e28-2061-4325-ae70-980c86ae72e8\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " Oct 30 17:24:26.425289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.425176 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-envoy\") pod \"a04e6e28-2061-4325-ae70-980c86ae72e8\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " Oct 30 17:24:26.425289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.425208 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-kube-api-access-5fj7b\" (UniqueName: \"kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-kube-api-access-5fj7b\") pod \"a04e6e28-2061-4325-ae70-980c86ae72e8\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " Oct 30 17:24:26.425289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.425280 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-podinfo\") pod \"a04e6e28-2061-4325-ae70-980c86ae72e8\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " Oct 30 17:24:26.425569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.425326 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a04e6e28-2061-4325-ae70-980c86ae72e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04e6e28-2061-4325-ae70-980c86ae72e8-istiod-ca-cert\") pod \"a04e6e28-2061-4325-ae70-980c86ae72e8\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " Oct 30 17:24:26.425569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.425437 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-certs\") pod \"a04e6e28-2061-4325-ae70-980c86ae72e8\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " Oct 30 17:24:26.425569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.425477 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-token\" (UniqueName: \"kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-token\") pod \"a04e6e28-2061-4325-ae70-980c86ae72e8\" (UID: \"a04e6e28-2061-4325-ae70-980c86ae72e8\") " Oct 30 17:24:26.425569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.425485 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-socket" pod "a04e6e28-2061-4325-ae70-980c86ae72e8" (UID: "a04e6e28-2061-4325-ae70-980c86ae72e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:26.425814 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.425578 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-credential-socket" pod "a04e6e28-2061-4325-ae70-980c86ae72e8" (UID: "a04e6e28-2061-4325-ae70-980c86ae72e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:26.426282 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.425886 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-certs" pod "a04e6e28-2061-4325-ae70-980c86ae72e8" (UID: "a04e6e28-2061-4325-ae70-980c86ae72e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:26.426413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.426331 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a04e6e28-2061-4325-ae70-980c86ae72e8-istiod-ca-cert" pod "a04e6e28-2061-4325-ae70-980c86ae72e8" (UID: "a04e6e28-2061-4325-ae70-980c86ae72e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:26.426487 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.426454 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.426487 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.426479 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.426609 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.426494 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.426609 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.426508 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a04e6e28-2061-4325-ae70-980c86ae72e8-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.426797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.426693 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-data" pod "a04e6e28-2061-4325-ae70-980c86ae72e8" (UID: "a04e6e28-2061-4325-ae70-980c86ae72e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:26.428412 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.428375 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-token" pod "a04e6e28-2061-4325-ae70-980c86ae72e8" (UID: "a04e6e28-2061-4325-ae70-980c86ae72e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:26.428569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.428549 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-podinfo" pod "a04e6e28-2061-4325-ae70-980c86ae72e8" (UID: "a04e6e28-2061-4325-ae70-980c86ae72e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:24:26.429959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.429887 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-envoy" pod "a04e6e28-2061-4325-ae70-980c86ae72e8" (UID: "a04e6e28-2061-4325-ae70-980c86ae72e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:26.430326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.430299 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-kube-api-access-5fj7b" pod "a04e6e28-2061-4325-ae70-980c86ae72e8" (UID: "a04e6e28-2061-4325-ae70-980c86ae72e8"). InnerVolumeSpecName "kube-api-access-5fj7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:26.467516 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:24:26.467480 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab756a2_144f_4411_85db_dcb56009529f.slice/crio-a50667ac465df18dcbfdee746f246bd8d0bb0a7c90061b17b1596e7a8dc8c968 WatchSource:0}: Error finding container a50667ac465df18dcbfdee746f246bd8d0bb0a7c90061b17b1596e7a8dc8c968: Status 404 returned error can't find the container with id a50667ac465df18dcbfdee746f246bd8d0bb0a7c90061b17b1596e7a8dc8c968 Oct 30 17:24:26.468797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.468772 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn"] Oct 30 17:24:26.470837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.470794 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:26.470953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.470889 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:26.471001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.470949 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:26.493775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.493725 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" podUID="55c3138d-b28b-4d4f-9173-48d05241873f" containerName="payload-pre-processing" containerID="cri-o://609a7fe64677fa57d3787e20b3e6f99785bbae7b05c3cf4f772aaf6550957287" gracePeriod=25 Oct 30 17:24:26.527444 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.527418 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-data\") pod \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " Oct 30 17:24:26.528399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.527575 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istiod-ca-cert\") pod \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " Oct 30 17:24:26.528399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.527640 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-certs\") pod \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " Oct 30 17:24:26.528399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.527718 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-data" pod "61ba7cfe-7c22-4bf6-97f8-ce90c9269462" (UID: "61ba7cfe-7c22-4bf6-97f8-ce90c9269462"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:26.528399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.527783 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-credential-socket\") pod \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " Oct 30 17:24:26.528399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.527811 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-podinfo\") pod \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " Oct 30 17:24:26.528399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.527861 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-socket\") pod \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " Oct 30 17:24:26.528399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.527928 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-kube-api-access-2gkqg\" (UniqueName: \"kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-kube-api-access-2gkqg\") pod \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " Oct 30 17:24:26.528399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.527995 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-token\" (UniqueName: \"kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-token\") pod \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " Oct 30 17:24:26.528399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.528018 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-certs" pod "61ba7cfe-7c22-4bf6-97f8-ce90c9269462" (UID: "61ba7cfe-7c22-4bf6-97f8-ce90c9269462"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:26.528399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.528145 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istiod-ca-cert" pod "61ba7cfe-7c22-4bf6-97f8-ce90c9269462" (UID: "61ba7cfe-7c22-4bf6-97f8-ce90c9269462"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:26.529434 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.528445 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-socket" pod "61ba7cfe-7c22-4bf6-97f8-ce90c9269462" (UID: "61ba7cfe-7c22-4bf6-97f8-ce90c9269462"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:26.529434 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.528667 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-credential-socket" pod "61ba7cfe-7c22-4bf6-97f8-ce90c9269462" (UID: "61ba7cfe-7c22-4bf6-97f8-ce90c9269462"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:26.529434 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.528787 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-envoy\") pod \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\" (UID: \"61ba7cfe-7c22-4bf6-97f8-ce90c9269462\") " Oct 30 17:24:26.529758 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.529719 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.529758 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.529756 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.529879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.529769 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.529879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.529782 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fj7b\" (UniqueName: \"kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-kube-api-access-5fj7b\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.529879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.529795 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.529879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.529809 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.529879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.529822 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.529879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.529839 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.529879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.529853 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.529879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.529869 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a04e6e28-2061-4325-ae70-980c86ae72e8-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.531116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.531061 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-token" pod "61ba7cfe-7c22-4bf6-97f8-ce90c9269462" (UID: "61ba7cfe-7c22-4bf6-97f8-ce90c9269462"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:26.534171 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.534090 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-envoy" pod "61ba7cfe-7c22-4bf6-97f8-ce90c9269462" (UID: "61ba7cfe-7c22-4bf6-97f8-ce90c9269462"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:26.534171 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.534138 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-kube-api-access-2gkqg" pod "61ba7cfe-7c22-4bf6-97f8-ce90c9269462" (UID: "61ba7cfe-7c22-4bf6-97f8-ce90c9269462"). InnerVolumeSpecName "kube-api-access-2gkqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:26.542386 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.542350 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-podinfo" pod "61ba7cfe-7c22-4bf6-97f8-ce90c9269462" (UID: "61ba7cfe-7c22-4bf6-97f8-ce90c9269462"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:24:26.591408 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.591310 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" podUID="0d8f899f-225a-4a7a-a938-41c23e83291e" containerName="main" containerID="cri-o://90e6b393b527a3d50204e5756363162fdf43816a26dee50aedc6181c4e70f095" gracePeriod=45 Oct 30 17:24:26.632227 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.632190 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.632227 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.632228 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gkqg\" (UniqueName: \"kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-kube-api-access-2gkqg\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.632456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.632244 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.632456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.632258 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ba7cfe-7c22-4bf6-97f8-ce90c9269462-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.924694 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.924385 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:24:26.949401 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.949366 4828 generic.go:356] "Generic (PLEG): container finished" podID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerID="a039b243aa94e9460182d4eeddd0a396c1c12a3f6ad8ac66b8698fb35bf639df" exitCode=0 Oct 30 17:24:26.949554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.949421 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" Oct 30 17:24:26.949554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.949465 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" event={"ID":"a04e6e28-2061-4325-ae70-980c86ae72e8","Type":"ContainerDied","Data":"a039b243aa94e9460182d4eeddd0a396c1c12a3f6ad8ac66b8698fb35bf639df"} Oct 30 17:24:26.950065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.950033 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p" event={"ID":"a04e6e28-2061-4325-ae70-980c86ae72e8","Type":"ContainerDied","Data":"147fc0bc45051cb0b36a0ca2a85316b536ba7fe6c56f659967f812445379e5ab"} Oct 30 17:24:26.950209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.950111 4828 scope.go:122] "RemoveContainer" containerID="a039b243aa94e9460182d4eeddd0a396c1c12a3f6ad8ac66b8698fb35bf639df" Oct 30 17:24:26.965367 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.965204 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" event={"ID":"7ab756a2-144f-4411-85db-dcb56009529f","Type":"ContainerStarted","Data":"5d2e9d2d0d1e8a3e6bfcb2dd13913477bee18a50ccbcb129da9689e4b660d326"} Oct 30 17:24:26.965367 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.965309 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" event={"ID":"7ab756a2-144f-4411-85db-dcb56009529f","Type":"ContainerStarted","Data":"a50667ac465df18dcbfdee746f246bd8d0bb0a7c90061b17b1596e7a8dc8c968"} Oct 30 17:24:26.967925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.967523 4828 generic.go:356] "Generic (PLEG): container finished" podID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerID="fa3d01ee6f657022887cfd7e37e34388db433fb66a84cf043d89e24ebcf40a45" exitCode=0 Oct 30 17:24:26.969656 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.967714 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" Oct 30 17:24:26.971419 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.967755 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" event={"ID":"61ba7cfe-7c22-4bf6-97f8-ce90c9269462","Type":"ContainerDied","Data":"fa3d01ee6f657022887cfd7e37e34388db433fb66a84cf043d89e24ebcf40a45"} Oct 30 17:24:26.971419 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.970649 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8" event={"ID":"61ba7cfe-7c22-4bf6-97f8-ce90c9269462","Type":"ContainerDied","Data":"f653738a5e05841b6a3beb157e81e3f5e5eefffad30516aca0b01ca1e74147de"} Oct 30 17:24:26.972220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.972194 4828 generic.go:356] "Generic (PLEG): container finished" podID="0d8f899f-225a-4a7a-a938-41c23e83291e" containerID="90e6b393b527a3d50204e5756363162fdf43816a26dee50aedc6181c4e70f095" exitCode=0 Oct 30 17:24:26.972402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.972385 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" event={"ID":"0d8f899f-225a-4a7a-a938-41c23e83291e","Type":"ContainerDied","Data":"90e6b393b527a3d50204e5756363162fdf43816a26dee50aedc6181c4e70f095"} Oct 30 17:24:26.972548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.972530 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" event={"ID":"0d8f899f-225a-4a7a-a938-41c23e83291e","Type":"ContainerDied","Data":"760d090410ac12b283f11cf3e4b159b7e2911721a01802c035141e174995a888"} Oct 30 17:24:26.972726 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.972712 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km" Oct 30 17:24:26.987636 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.987609 4828 scope.go:122] "RemoveContainer" containerID="a039b243aa94e9460182d4eeddd0a396c1c12a3f6ad8ac66b8698fb35bf639df" Oct 30 17:24:26.988230 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:24:26.988159 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a039b243aa94e9460182d4eeddd0a396c1c12a3f6ad8ac66b8698fb35bf639df\": container with ID starting with a039b243aa94e9460182d4eeddd0a396c1c12a3f6ad8ac66b8698fb35bf639df not found: ID does not exist" containerID="a039b243aa94e9460182d4eeddd0a396c1c12a3f6ad8ac66b8698fb35bf639df" Oct 30 17:24:26.988409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.988375 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a039b243aa94e9460182d4eeddd0a396c1c12a3f6ad8ac66b8698fb35bf639df"} err="failed to get container status \"a039b243aa94e9460182d4eeddd0a396c1c12a3f6ad8ac66b8698fb35bf639df\": rpc error: code = NotFound desc = could not find container \"a039b243aa94e9460182d4eeddd0a396c1c12a3f6ad8ac66b8698fb35bf639df\": container with ID starting with a039b243aa94e9460182d4eeddd0a396c1c12a3f6ad8ac66b8698fb35bf639df not found: ID does not exist" Oct 30 17:24:26.988530 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:26.988411 4828 scope.go:122] "RemoveContainer" containerID="fa3d01ee6f657022887cfd7e37e34388db433fb66a84cf043d89e24ebcf40a45" Oct 30 17:24:27.005246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.005208 4828 scope.go:122] "RemoveContainer" containerID="fa3d01ee6f657022887cfd7e37e34388db433fb66a84cf043d89e24ebcf40a45" Oct 30 17:24:27.012586 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:24:27.012439 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3d01ee6f657022887cfd7e37e34388db433fb66a84cf043d89e24ebcf40a45\": container with ID starting with fa3d01ee6f657022887cfd7e37e34388db433fb66a84cf043d89e24ebcf40a45 not found: ID does not exist" containerID="fa3d01ee6f657022887cfd7e37e34388db433fb66a84cf043d89e24ebcf40a45" Oct 30 17:24:27.012586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.012480 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3d01ee6f657022887cfd7e37e34388db433fb66a84cf043d89e24ebcf40a45"} err="failed to get container status \"fa3d01ee6f657022887cfd7e37e34388db433fb66a84cf043d89e24ebcf40a45\": rpc error: code = NotFound desc = could not find container \"fa3d01ee6f657022887cfd7e37e34388db433fb66a84cf043d89e24ebcf40a45\": container with ID starting with fa3d01ee6f657022887cfd7e37e34388db433fb66a84cf043d89e24ebcf40a45 not found: ID does not exist" Oct 30 17:24:27.012586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.012507 4828 scope.go:122] "RemoveContainer" containerID="90e6b393b527a3d50204e5756363162fdf43816a26dee50aedc6181c4e70f095" Oct 30 17:24:27.016405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.016382 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p"] Oct 30 17:24:27.024261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.024220 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6j56p"] Oct 30 17:24:27.035148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.035000 4828 scope.go:122] "RemoveContainer" containerID="90e6b393b527a3d50204e5756363162fdf43816a26dee50aedc6181c4e70f095" Oct 30 17:24:27.035444 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:24:27.035413 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e6b393b527a3d50204e5756363162fdf43816a26dee50aedc6181c4e70f095\": container with ID starting with 90e6b393b527a3d50204e5756363162fdf43816a26dee50aedc6181c4e70f095 not found: ID does not exist" containerID="90e6b393b527a3d50204e5756363162fdf43816a26dee50aedc6181c4e70f095" Oct 30 17:24:27.035634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.035602 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e6b393b527a3d50204e5756363162fdf43816a26dee50aedc6181c4e70f095"} err="failed to get container status \"90e6b393b527a3d50204e5756363162fdf43816a26dee50aedc6181c4e70f095\": rpc error: code = NotFound desc = could not find container \"90e6b393b527a3d50204e5756363162fdf43816a26dee50aedc6181c4e70f095\": container with ID starting with 90e6b393b527a3d50204e5756363162fdf43816a26dee50aedc6181c4e70f095 not found: ID does not exist" Oct 30 17:24:27.044541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.042783 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-tmp-dir\") pod \"0d8f899f-225a-4a7a-a938-41c23e83291e\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " Oct 30 17:24:27.044541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.042998 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d8f899f-225a-4a7a-a938-41c23e83291e-kube-api-access-m6rq9\" (UniqueName: \"kubernetes.io/projected/0d8f899f-225a-4a7a-a938-41c23e83291e-kube-api-access-m6rq9\") pod \"0d8f899f-225a-4a7a-a938-41c23e83291e\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " Oct 30 17:24:27.044541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.043043 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-model-cache\") pod \"0d8f899f-225a-4a7a-a938-41c23e83291e\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " Oct 30 17:24:27.044541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.043241 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d8f899f-225a-4a7a-a938-41c23e83291e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8f899f-225a-4a7a-a938-41c23e83291e-tls-certs\") pod \"0d8f899f-225a-4a7a-a938-41c23e83291e\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " Oct 30 17:24:27.044541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.043286 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-home\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-home\") pod \"0d8f899f-225a-4a7a-a938-41c23e83291e\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " Oct 30 17:24:27.044541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.043319 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-dshm\") pod \"0d8f899f-225a-4a7a-a938-41c23e83291e\" (UID: \"0d8f899f-225a-4a7a-a938-41c23e83291e\") " Oct 30 17:24:27.045459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.045118 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-tmp-dir" pod "0d8f899f-225a-4a7a-a938-41c23e83291e" (UID: "0d8f899f-225a-4a7a-a938-41c23e83291e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:27.047984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.046323 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-model-cache" pod "0d8f899f-225a-4a7a-a938-41c23e83291e" (UID: "0d8f899f-225a-4a7a-a938-41c23e83291e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:27.063548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.051254 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-home" pod "0d8f899f-225a-4a7a-a938-41c23e83291e" (UID: "0d8f899f-225a-4a7a-a938-41c23e83291e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:27.077291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.077234 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8f899f-225a-4a7a-a938-41c23e83291e-tls-certs" pod "0d8f899f-225a-4a7a-a938-41c23e83291e" (UID: "0d8f899f-225a-4a7a-a938-41c23e83291e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:27.095820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.095763 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" podStartSLOduration=2.095743107 podStartE2EDuration="2.095743107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:27.065955266 +0000 UTC m=+2244.711465448" watchObservedRunningTime="2026-10-30 17:24:27.095743107 +0000 UTC m=+2244.741253288" Oct 30 17:24:27.096848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.096818 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb"] Oct 30 17:24:27.098219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.098195 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d8f899f-225a-4a7a-a938-41c23e83291e-kube-api-access-m6rq9" pod "0d8f899f-225a-4a7a-a938-41c23e83291e" (UID: "0d8f899f-225a-4a7a-a938-41c23e83291e"). InnerVolumeSpecName "kube-api-access-m6rq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:27.101015 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.100986 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-dshm" pod "0d8f899f-225a-4a7a-a938-41c23e83291e" (UID: "0d8f899f-225a-4a7a-a938-41c23e83291e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:27.125764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.125709 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:27.126592 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.126542 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 17:24:27.126749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.126603 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" podUID="7ab756a2-144f-4411-85db-dcb56009529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 17:24:27.145061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.144977 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6rq9\" (UniqueName: \"kubernetes.io/projected/0d8f899f-225a-4a7a-a938-41c23e83291e-kube-api-access-m6rq9\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:27.145061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.145007 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:27.145061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.145018 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d8f899f-225a-4a7a-a938-41c23e83291e-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:27.145061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.145026 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:27.145061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.145033 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:27.145061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.145040 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d8f899f-225a-4a7a-a938-41c23e83291e-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:27.150395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.150339 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8"] Oct 30 17:24:27.157839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.157796 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-n8mh8"] Oct 30 17:24:27.165341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.165264 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:27.413807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.413762 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km"] Oct 30 17:24:27.418947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:27.418849 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e5ac8f/test-model-e5ac8f-kserve-69658fc9d7-826km"] Oct 30 17:24:28.101570 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:28.101530 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:28.114039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:28.114011 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 17:24:28.114203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:28.114064 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" podUID="7ab756a2-144f-4411-85db-dcb56009529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 17:24:28.359280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:28.359184 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:28.549927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:28.543629 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:28.558951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:28.550574 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d8f899f-225a-4a7a-a938-41c23e83291e" path="/var/lib/kubelet/pods/0d8f899f-225a-4a7a-a938-41c23e83291e/volumes" Oct 30 17:24:28.558951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:28.551452 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" path="/var/lib/kubelet/pods/61ba7cfe-7c22-4bf6-97f8-ce90c9269462/volumes" Oct 30 17:24:28.558951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:28.552258 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" path="/var/lib/kubelet/pods/a04e6e28-2061-4325-ae70-980c86ae72e8/volumes" Oct 30 17:24:28.811255 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:28.810552 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:29.127554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:29.127457 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 17:24:29.127554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:29.127511 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" podUID="7ab756a2-144f-4411-85db-dcb56009529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 17:24:30.088424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:30.088383 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:30.117410 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:30.117361 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 17:24:30.121659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:30.121618 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" podUID="7ab756a2-144f-4411-85db-dcb56009529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 17:24:31.114863 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:31.114824 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 17:24:31.115394 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:31.114891 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" podUID="7ab756a2-144f-4411-85db-dcb56009529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 17:24:31.780662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:31.780619 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:32.114058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:32.113970 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 17:24:32.114203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:32.114046 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" podUID="7ab756a2-144f-4411-85db-dcb56009529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 17:24:33.114290 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:33.114253 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 17:24:33.114704 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:33.114324 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" podUID="7ab756a2-144f-4411-85db-dcb56009529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 17:24:33.185118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:33.185082 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv"] Oct 30 17:24:33.185334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:33.185311 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" podUID="8f31a368-af2f-417f-8863-5ab835b7995f" containerName="maas-api" containerID="cri-o://d1b2e9066ce1630ef944e2bbedb8537ecee6b43d0154626315df16b8d5d1d1e8" gracePeriod=30 Oct 30 17:24:33.281630 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:33.281580 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:33.899769 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:33.897496 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:33.940657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:33.940606 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:24:34.056986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.056947 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f31a368-af2f-417f-8863-5ab835b7995f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f31a368-af2f-417f-8863-5ab835b7995f-openshift-service-ca\") pod \"8f31a368-af2f-417f-8863-5ab835b7995f\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " Oct 30 17:24:34.057204 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.057071 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-metrics-certs\") pod \"8f31a368-af2f-417f-8863-5ab835b7995f\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " Oct 30 17:24:34.057204 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.057115 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-maas-api-tls\") pod \"8f31a368-af2f-417f-8863-5ab835b7995f\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " Oct 30 17:24:34.057204 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.057175 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f31a368-af2f-417f-8863-5ab835b7995f-kube-api-access-w75gh\" (UniqueName: \"kubernetes.io/projected/8f31a368-af2f-417f-8863-5ab835b7995f-kube-api-access-w75gh\") pod \"8f31a368-af2f-417f-8863-5ab835b7995f\" (UID: \"8f31a368-af2f-417f-8863-5ab835b7995f\") " Oct 30 17:24:34.057428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.057398 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f31a368-af2f-417f-8863-5ab835b7995f-openshift-service-ca" pod "8f31a368-af2f-417f-8863-5ab835b7995f" (UID: "8f31a368-af2f-417f-8863-5ab835b7995f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:34.057783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.057754 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f31a368-af2f-417f-8863-5ab835b7995f-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:34.059489 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.059464 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-metrics-certs" pod "8f31a368-af2f-417f-8863-5ab835b7995f" (UID: "8f31a368-af2f-417f-8863-5ab835b7995f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:34.059827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.059802 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-maas-api-tls" pod "8f31a368-af2f-417f-8863-5ab835b7995f" (UID: "8f31a368-af2f-417f-8863-5ab835b7995f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:34.059941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.059824 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f31a368-af2f-417f-8863-5ab835b7995f-kube-api-access-w75gh" pod "8f31a368-af2f-417f-8863-5ab835b7995f" (UID: "8f31a368-af2f-417f-8863-5ab835b7995f"). InnerVolumeSpecName "kube-api-access-w75gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:34.120135 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.120103 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:34.120729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.120701 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:34.121622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.121602 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:24:34.155953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.145302 4828 generic.go:356] "Generic (PLEG): container finished" podID="8f31a368-af2f-417f-8863-5ab835b7995f" containerID="d1b2e9066ce1630ef944e2bbedb8537ecee6b43d0154626315df16b8d5d1d1e8" exitCode=0 Oct 30 17:24:34.155953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.146509 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" Oct 30 17:24:34.159935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.157077 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" event={"ID":"8f31a368-af2f-417f-8863-5ab835b7995f","Type":"ContainerDied","Data":"d1b2e9066ce1630ef944e2bbedb8537ecee6b43d0154626315df16b8d5d1d1e8"} Oct 30 17:24:34.159935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.157164 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv" event={"ID":"8f31a368-af2f-417f-8863-5ab835b7995f","Type":"ContainerDied","Data":"711f8d1a4ee1ac05deb56bd0963df715e415961beca8d61aeba0ee39743f9e7b"} Oct 30 17:24:34.159935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.157198 4828 scope.go:122] "RemoveContainer" containerID="d1b2e9066ce1630ef944e2bbedb8537ecee6b43d0154626315df16b8d5d1d1e8" Oct 30 17:24:34.161799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.161668 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:34.161799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.161705 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f31a368-af2f-417f-8863-5ab835b7995f-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:34.161799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.161721 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w75gh\" (UniqueName: \"kubernetes.io/projected/8f31a368-af2f-417f-8863-5ab835b7995f-kube-api-access-w75gh\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:34.171931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.164403 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:34.213419 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.212890 4828 scope.go:122] "RemoveContainer" containerID="d1b2e9066ce1630ef944e2bbedb8537ecee6b43d0154626315df16b8d5d1d1e8" Oct 30 17:24:34.213626 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:24:34.213552 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b2e9066ce1630ef944e2bbedb8537ecee6b43d0154626315df16b8d5d1d1e8\": container with ID starting with d1b2e9066ce1630ef944e2bbedb8537ecee6b43d0154626315df16b8d5d1d1e8 not found: ID does not exist" containerID="d1b2e9066ce1630ef944e2bbedb8537ecee6b43d0154626315df16b8d5d1d1e8" Oct 30 17:24:34.213626 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.213594 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b2e9066ce1630ef944e2bbedb8537ecee6b43d0154626315df16b8d5d1d1e8"} err="failed to get container status \"d1b2e9066ce1630ef944e2bbedb8537ecee6b43d0154626315df16b8d5d1d1e8\": rpc error: code = NotFound desc = could not find container \"d1b2e9066ce1630ef944e2bbedb8537ecee6b43d0154626315df16b8d5d1d1e8\": container with ID starting with d1b2e9066ce1630ef944e2bbedb8537ecee6b43d0154626315df16b8d5d1d1e8 not found: ID does not exist" Oct 30 17:24:34.328472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.328385 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv"] Oct 30 17:24:34.333185 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.333150 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-2d01ec-74b65769dd-ffskv"] Oct 30 17:24:34.501923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.500028 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:34.560986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.559281 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f31a368-af2f-417f-8863-5ab835b7995f" path="/var/lib/kubelet/pods/8f31a368-af2f-417f-8863-5ab835b7995f/volumes" Oct 30 17:24:34.744561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.744470 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" podUID="55c3138d-b28b-4d4f-9173-48d05241873f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:24:34.803830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:34.803791 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:36.423924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:36.413862 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:37.912835 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:24:37.912798 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:24:38.000925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:37.997389 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:39.837940 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:39.828647 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:40.819186 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.819145 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7"] Oct 30 17:24:40.819967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.819951 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" Oct 30 17:24:40.820023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.819967 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" Oct 30 17:24:40.820023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.819991 4828 cpu_manager.go:414] "containerMap: removing container" podUID="8f31a368-af2f-417f-8863-5ab835b7995f" containerName="maas-api" Oct 30 17:24:40.820023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.819997 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f31a368-af2f-417f-8863-5ab835b7995f" containerName="maas-api" Oct 30 17:24:40.820023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.820017 4828 cpu_manager.go:414] "containerMap: removing container" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" Oct 30 17:24:40.820023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.820022 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" Oct 30 17:24:40.820183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.820036 4828 cpu_manager.go:414] "containerMap: removing container" podUID="0d8f899f-225a-4a7a-a938-41c23e83291e" containerName="main" Oct 30 17:24:40.820183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.820041 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d8f899f-225a-4a7a-a938-41c23e83291e" containerName="main" Oct 30 17:24:40.820183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.820156 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a04e6e28-2061-4325-ae70-980c86ae72e8" containerName="istio-proxy" Oct 30 17:24:40.820183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.820167 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f31a368-af2f-417f-8863-5ab835b7995f" containerName="maas-api" Oct 30 17:24:40.820183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.820177 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ba7cfe-7c22-4bf6-97f8-ce90c9269462" containerName="istio-proxy" Oct 30 17:24:40.820374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.820185 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d8f899f-225a-4a7a-a938-41c23e83291e" containerName="main" Oct 30 17:24:40.825828 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.825787 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:40.837506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.837477 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6"] Oct 30 17:24:40.842820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.842797 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:40.850679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.850649 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7"] Oct 30 17:24:40.869693 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.869646 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6"] Oct 30 17:24:40.900349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.900310 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:40.900349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.900355 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:40.900664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.900387 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctz6k\" (UniqueName: \"kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-kube-api-access-ctz6k\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:40.900664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.900460 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:40.900664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.900499 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:40.900664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.900645 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:40.900869 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.900686 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25103151-7048-4796-adab-7e5b0b971e7a-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:40.900869 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.900799 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25103151-7048-4796-adab-7e5b0b971e7a-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:40.901021 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:40.901001 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.001990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.001955 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.002197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002000 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25103151-7048-4796-adab-7e5b0b971e7a-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.002197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002052 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25103151-7048-4796-adab-7e5b0b971e7a-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.002197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002076 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.002197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002130 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.002352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002241 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a0eb80-9692-4e67-a75b-43f638a8177c-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.002352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002334 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.002443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002415 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.002478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002451 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.002705 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002674 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.002856 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002722 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.002856 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002758 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctz6k\" (UniqueName: \"kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-kube-api-access-ctz6k\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.002856 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002809 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.002856 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002824 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.002856 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002838 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.003173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.002956 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.003173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.003038 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.003173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.003073 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvs52\" (UniqueName: \"kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-kube-api-access-nvs52\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.003173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.003093 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.003173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.003113 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.003509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.003216 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25103151-7048-4796-adab-7e5b0b971e7a-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.003509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.003488 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.003678 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.003655 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.004773 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.004748 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.005665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.005643 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25103151-7048-4796-adab-7e5b0b971e7a-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.026562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.026527 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.026778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.026754 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctz6k\" (UniqueName: \"kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-kube-api-access-ctz6k\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.105197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.105101 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.105197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.105141 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.105425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.105198 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.105425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.105271 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.105425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.105300 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvs52\" (UniqueName: \"kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-kube-api-access-nvs52\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.105425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.105347 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.105634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.105481 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.105634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.105549 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.105754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.105636 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a0eb80-9692-4e67-a75b-43f638a8177c-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.105754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.105730 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.105959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.105894 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-data\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.106181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.106159 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-credential-socket\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.106250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.106225 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-certs\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.106637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.106571 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a0eb80-9692-4e67-a75b-43f638a8177c-istiod-ca-cert\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.141646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.141610 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:41.200687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.200648 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-envoy\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.201871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.200776 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-podinfo\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.201871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.200978 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvs52\" (UniqueName: \"kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-kube-api-access-nvs52\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.201871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.201611 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-token\") pod \"e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.387299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.387177 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:41.472927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.465168 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:41.541756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.541610 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:41.595168 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:24:41.595118 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25103151_7048_4796_adab_7e5b0b971e7a.slice/crio-c57152bbc73767eb13aa1eb04f6b549ddb9ea759763373c3636502b01df7b84d WatchSource:0}: Error finding container c57152bbc73767eb13aa1eb04f6b549ddb9ea759763373c3636502b01df7b84d: Status 404 returned error can't find the container with id c57152bbc73767eb13aa1eb04f6b549ddb9ea759763373c3636502b01df7b84d Oct 30 17:24:41.599346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.599266 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:41.599459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.599404 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:41.599459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.599448 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:41.600433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.600124 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7"] Oct 30 17:24:41.700787 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.700754 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7"] Oct 30 17:24:41.701362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.701001 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" containerID="cri-o://3542f6d908e1a4ffffae1d8a8b73c1532976292cb19fcae70583f4438485a602" gracePeriod=30 Oct 30 17:24:41.761563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:41.761525 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:42.030756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.027779 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6"] Oct 30 17:24:42.030756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.029709 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:42.030756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.030000 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:42.030756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.030116 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:24:42.137019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.136986 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" podUID="b1a87552-a6e2-451c-bd7c-e8b781c48aa4" containerName="main" containerID="cri-o://fcdfe6b5498ee744b0e80b4bba88bb681f2b694e1dc93b8411fd1a7efd6ef9aa" gracePeriod=45 Oct 30 17:24:42.238092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.238002 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" event={"ID":"25103151-7048-4796-adab-7e5b0b971e7a","Type":"ContainerStarted","Data":"1a3c0bafc5b2e01c43d4a68e21febaa04248ff235243c483bd4b045fae241a24"} Oct 30 17:24:42.238092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.238086 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" event={"ID":"25103151-7048-4796-adab-7e5b0b971e7a","Type":"ContainerStarted","Data":"c57152bbc73767eb13aa1eb04f6b549ddb9ea759763373c3636502b01df7b84d"} Oct 30 17:24:42.240402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.240368 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" event={"ID":"44a0eb80-9692-4e67-a75b-43f638a8177c","Type":"ContainerStarted","Data":"9d23f01431dad247dbf0396b5269a8f0ded1390ad470a33c6413c1e3c155071a"} Oct 30 17:24:42.341338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.341249 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" podStartSLOduration=2.3412349519999998 podStartE2EDuration="2.341234952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:42.336339786 +0000 UTC m=+2259.981849965" watchObservedRunningTime="2026-10-30 17:24:42.341234952 +0000 UTC m=+2259.986745131" Oct 30 17:24:42.531960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.531919 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:24:42.672587 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.672551 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-home\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-home\") pod \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " Oct 30 17:24:42.672805 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.672725 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-dshm\") pod \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " Oct 30 17:24:42.672805 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.672787 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tmp-dir\") pod \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " Oct 30 17:24:42.672940 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.672817 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-model-cache\") pod \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " Oct 30 17:24:42.672940 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.672880 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tls-certs\") pod \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " Oct 30 17:24:42.673064 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.673018 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-kube-api-access-hkbpf\" (UniqueName: \"kubernetes.io/projected/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-kube-api-access-hkbpf\") pod \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\" (UID: \"b1a87552-a6e2-451c-bd7c-e8b781c48aa4\") " Oct 30 17:24:42.673606 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.673571 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-home" pod "b1a87552-a6e2-451c-bd7c-e8b781c48aa4" (UID: "b1a87552-a6e2-451c-bd7c-e8b781c48aa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.673800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.673779 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-model-cache" pod "b1a87552-a6e2-451c-bd7c-e8b781c48aa4" (UID: "b1a87552-a6e2-451c-bd7c-e8b781c48aa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.674028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.674005 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tmp-dir" pod "b1a87552-a6e2-451c-bd7c-e8b781c48aa4" (UID: "b1a87552-a6e2-451c-bd7c-e8b781c48aa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.674727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.674701 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.674890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.674872 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.675057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.675042 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.675800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.675775 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-dshm" pod "b1a87552-a6e2-451c-bd7c-e8b781c48aa4" (UID: "b1a87552-a6e2-451c-bd7c-e8b781c48aa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.676963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.676889 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-kube-api-access-hkbpf" pod "b1a87552-a6e2-451c-bd7c-e8b781c48aa4" (UID: "b1a87552-a6e2-451c-bd7c-e8b781c48aa4"). InnerVolumeSpecName "kube-api-access-hkbpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:42.682515 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.682486 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tls-certs" pod "b1a87552-a6e2-451c-bd7c-e8b781c48aa4" (UID: "b1a87552-a6e2-451c-bd7c-e8b781c48aa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:42.777558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.777506 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.777558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.777550 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.777558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:42.777567 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkbpf\" (UniqueName: \"kubernetes.io/projected/b1a87552-a6e2-451c-bd7c-e8b781c48aa4-kube-api-access-hkbpf\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:43.145040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.145004 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:43.146014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.145983 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 17:24:43.146222 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.146197 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" podUID="25103151-7048-4796-adab-7e5b0b971e7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 17:24:43.251011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.250971 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" event={"ID":"44a0eb80-9692-4e67-a75b-43f638a8177c","Type":"ContainerStarted","Data":"63ec38b2f9d0b6ff3d59c68c40f1739487206e70817597bf63a8d60949b8c8e8"} Oct 30 17:24:43.252766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.252742 4828 generic.go:356] "Generic (PLEG): container finished" podID="b1a87552-a6e2-451c-bd7c-e8b781c48aa4" containerID="fcdfe6b5498ee744b0e80b4bba88bb681f2b694e1dc93b8411fd1a7efd6ef9aa" exitCode=0 Oct 30 17:24:43.252941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.252792 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" Oct 30 17:24:43.252941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.252822 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" event={"ID":"b1a87552-a6e2-451c-bd7c-e8b781c48aa4","Type":"ContainerDied","Data":"fcdfe6b5498ee744b0e80b4bba88bb681f2b694e1dc93b8411fd1a7efd6ef9aa"} Oct 30 17:24:43.252941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.252857 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" event={"ID":"b1a87552-a6e2-451c-bd7c-e8b781c48aa4","Type":"ContainerDied","Data":"d0a141bdb69f5c81d228815f3046f9504914c0b2878f82a114eed6a494aca7d1"} Oct 30 17:24:43.252941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.252873 4828 scope.go:122] "RemoveContainer" containerID="fcdfe6b5498ee744b0e80b4bba88bb681f2b694e1dc93b8411fd1a7efd6ef9aa" Oct 30 17:24:43.266000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.265976 4828 scope.go:122] "RemoveContainer" containerID="fcdfe6b5498ee744b0e80b4bba88bb681f2b694e1dc93b8411fd1a7efd6ef9aa" Oct 30 17:24:43.266274 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:24:43.266250 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcdfe6b5498ee744b0e80b4bba88bb681f2b694e1dc93b8411fd1a7efd6ef9aa\": container with ID starting with fcdfe6b5498ee744b0e80b4bba88bb681f2b694e1dc93b8411fd1a7efd6ef9aa not found: ID does not exist" containerID="fcdfe6b5498ee744b0e80b4bba88bb681f2b694e1dc93b8411fd1a7efd6ef9aa" Oct 30 17:24:43.266349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.266284 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcdfe6b5498ee744b0e80b4bba88bb681f2b694e1dc93b8411fd1a7efd6ef9aa"} err="failed to get container status \"fcdfe6b5498ee744b0e80b4bba88bb681f2b694e1dc93b8411fd1a7efd6ef9aa\": rpc error: code = NotFound desc = could not find container \"fcdfe6b5498ee744b0e80b4bba88bb681f2b694e1dc93b8411fd1a7efd6ef9aa\": container with ID starting with fcdfe6b5498ee744b0e80b4bba88bb681f2b694e1dc93b8411fd1a7efd6ef9aa not found: ID does not exist" Oct 30 17:24:43.325982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.320355 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" podStartSLOduration=3.320333152 podStartE2EDuration="3.320333152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:43.299333844 +0000 UTC m=+2260.944844025" watchObservedRunningTime="2026-10-30 17:24:43.320333152 +0000 UTC m=+2260.965843333" Oct 30 17:24:43.346891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.346854 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb"] Oct 30 17:24:43.350306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.350282 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb"] Oct 30 17:24:43.462625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.462586 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-42d381/test-model-42d381-kserve-5d5bbb5f79-nl6kb" podUID="b1a87552-a6e2-451c-bd7c-e8b781c48aa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.182:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:24:43.467207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.467179 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:43.468858 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.468841 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 17:24:43.468964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.468878 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" podUID="44a0eb80-9692-4e67-a75b-43f638a8177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 17:24:43.767779 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:43.764102 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:44.142972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:44.142330 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 17:24:44.142972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:44.142388 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" podUID="25103151-7048-4796-adab-7e5b0b971e7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 17:24:44.467312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:44.467220 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 17:24:44.467312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:44.467282 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" podUID="44a0eb80-9692-4e67-a75b-43f638a8177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 17:24:44.546452 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:44.546416 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a87552-a6e2-451c-bd7c-e8b781c48aa4" path="/var/lib/kubelet/pods/b1a87552-a6e2-451c-bd7c-e8b781c48aa4/volumes" Oct 30 17:24:44.743603 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:44.743523 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" podUID="55c3138d-b28b-4d4f-9173-48d05241873f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:24:45.142726 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:45.142687 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 17:24:45.142947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:45.142747 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" podUID="25103151-7048-4796-adab-7e5b0b971e7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 17:24:45.467517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:45.467432 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 17:24:45.467517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:45.467480 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" podUID="44a0eb80-9692-4e67-a75b-43f638a8177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 17:24:45.790925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:45.788485 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:46.142492 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:46.142378 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 17:24:46.142492 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:46.142464 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" podUID="25103151-7048-4796-adab-7e5b0b971e7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 17:24:46.468465 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:46.467344 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 17:24:46.468465 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:46.467408 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" podUID="44a0eb80-9692-4e67-a75b-43f638a8177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 17:24:47.257812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.257369 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 17:24:47.257812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.257428 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" podUID="25103151-7048-4796-adab-7e5b0b971e7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 17:24:47.368680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.368639 4828 generic.go:356] "Generic (PLEG): container finished" podID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerID="3542f6d908e1a4ffffae1d8a8b73c1532976292cb19fcae70583f4438485a602" exitCode=0 Oct 30 17:24:47.368887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.368812 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" event={"ID":"0d18657c-89fd-4f95-af19-096ef42f4e18","Type":"ContainerDied","Data":"3542f6d908e1a4ffffae1d8a8b73c1532976292cb19fcae70583f4438485a602"} Oct 30 17:24:47.368887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.368869 4828 scope.go:122] "RemoveContainer" containerID="8432fc75d285267ad575baa2b0a8632340a5e625971a1d58567182113bd726ed" Oct 30 17:24:47.415927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.403788 4828 generic.go:356] "Generic (PLEG): container finished" podID="55c3138d-b28b-4d4f-9173-48d05241873f" containerID="609a7fe64677fa57d3787e20b3e6f99785bbae7b05c3cf4f772aaf6550957287" exitCode=0 Oct 30 17:24:47.415927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.404079 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" event={"ID":"55c3138d-b28b-4d4f-9173-48d05241873f","Type":"ContainerDied","Data":"609a7fe64677fa57d3787e20b3e6f99785bbae7b05c3cf4f772aaf6550957287"} Oct 30 17:24:47.415927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.404140 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" event={"ID":"55c3138d-b28b-4d4f-9173-48d05241873f","Type":"ContainerDied","Data":"0c62f09cd76a3183ff340c40e4b193692fb15d2a74326d47cc5d69329891aed4"} Oct 30 17:24:47.415927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.404167 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c62f09cd76a3183ff340c40e4b193692fb15d2a74326d47cc5d69329891aed4" Oct 30 17:24:47.415927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.410433 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" Oct 30 17:24:47.458800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.458771 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c3138d-b28b-4d4f-9173-48d05241873f-kube-api-access-dlldr\" (UniqueName: \"kubernetes.io/projected/55c3138d-b28b-4d4f-9173-48d05241873f-kube-api-access-dlldr\") pod \"55c3138d-b28b-4d4f-9173-48d05241873f\" (UID: \"55c3138d-b28b-4d4f-9173-48d05241873f\") " Oct 30 17:24:47.458986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.458871 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c3138d-b28b-4d4f-9173-48d05241873f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3138d-b28b-4d4f-9173-48d05241873f-plugins-config-volume\") pod \"55c3138d-b28b-4d4f-9173-48d05241873f\" (UID: \"55c3138d-b28b-4d4f-9173-48d05241873f\") " Oct 30 17:24:47.460127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.460098 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c3138d-b28b-4d4f-9173-48d05241873f-plugins-config-volume" pod "55c3138d-b28b-4d4f-9173-48d05241873f" (UID: "55c3138d-b28b-4d4f-9173-48d05241873f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:47.462771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.462744 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c3138d-b28b-4d4f-9173-48d05241873f-kube-api-access-dlldr" pod "55c3138d-b28b-4d4f-9173-48d05241873f" (UID: "55c3138d-b28b-4d4f-9173-48d05241873f"). InnerVolumeSpecName "kube-api-access-dlldr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:47.468299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.468272 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 17:24:47.468439 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.468320 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" podUID="44a0eb80-9692-4e67-a75b-43f638a8177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 17:24:47.561966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.561933 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlldr\" (UniqueName: \"kubernetes.io/projected/55c3138d-b28b-4d4f-9173-48d05241873f-kube-api-access-dlldr\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:47.562357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.562020 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55c3138d-b28b-4d4f-9173-48d05241873f-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:47.635377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.633479 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:24:47.663956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.663390 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-socket\") pod \"0d18657c-89fd-4f95-af19-096ef42f4e18\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " Oct 30 17:24:47.663956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.663450 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-envoy\") pod \"0d18657c-89fd-4f95-af19-096ef42f4e18\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " Oct 30 17:24:47.663956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.663552 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-certs\") pod \"0d18657c-89fd-4f95-af19-096ef42f4e18\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " Oct 30 17:24:47.663956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.663767 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-podinfo\") pod \"0d18657c-89fd-4f95-af19-096ef42f4e18\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " Oct 30 17:24:47.663956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.663832 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-data\") pod \"0d18657c-89fd-4f95-af19-096ef42f4e18\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " Oct 30 17:24:47.663956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.663865 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-credential-socket\") pod \"0d18657c-89fd-4f95-af19-096ef42f4e18\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " Oct 30 17:24:47.663956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.663949 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-kube-api-access-2r629\" (UniqueName: \"kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-kube-api-access-2r629\") pod \"0d18657c-89fd-4f95-af19-096ef42f4e18\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " Oct 30 17:24:47.664499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.663980 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d18657c-89fd-4f95-af19-096ef42f4e18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d18657c-89fd-4f95-af19-096ef42f4e18-istiod-ca-cert\") pod \"0d18657c-89fd-4f95-af19-096ef42f4e18\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " Oct 30 17:24:47.664499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.664061 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-token\" (UniqueName: \"kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-token\") pod \"0d18657c-89fd-4f95-af19-096ef42f4e18\" (UID: \"0d18657c-89fd-4f95-af19-096ef42f4e18\") " Oct 30 17:24:47.664499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.664279 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-certs" pod "0d18657c-89fd-4f95-af19-096ef42f4e18" (UID: "0d18657c-89fd-4f95-af19-096ef42f4e18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:47.664724 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.664698 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-data" pod "0d18657c-89fd-4f95-af19-096ef42f4e18" (UID: "0d18657c-89fd-4f95-af19-096ef42f4e18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:47.664808 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.664759 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d18657c-89fd-4f95-af19-096ef42f4e18-istiod-ca-cert" pod "0d18657c-89fd-4f95-af19-096ef42f4e18" (UID: "0d18657c-89fd-4f95-af19-096ef42f4e18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:47.664926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.664893 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-credential-socket" pod "0d18657c-89fd-4f95-af19-096ef42f4e18" (UID: "0d18657c-89fd-4f95-af19-096ef42f4e18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:47.665142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.665108 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:47.665219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.665146 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:47.665219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.665158 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:47.665219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.665166 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d18657c-89fd-4f95-af19-096ef42f4e18-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:47.665318 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.665234 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-socket" pod "0d18657c-89fd-4f95-af19-096ef42f4e18" (UID: "0d18657c-89fd-4f95-af19-096ef42f4e18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:47.703925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.689101 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-podinfo" pod "0d18657c-89fd-4f95-af19-096ef42f4e18" (UID: "0d18657c-89fd-4f95-af19-096ef42f4e18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:24:47.703925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.692630 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-token" pod "0d18657c-89fd-4f95-af19-096ef42f4e18" (UID: "0d18657c-89fd-4f95-af19-096ef42f4e18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:47.704550 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.704455 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-envoy" pod "0d18657c-89fd-4f95-af19-096ef42f4e18" (UID: "0d18657c-89fd-4f95-af19-096ef42f4e18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:47.878938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.876835 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:47.878938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.876872 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:47.878938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.876886 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:47.878938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.876919 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d18657c-89fd-4f95-af19-096ef42f4e18-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:47.921853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:47.920446 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:48.056879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.056833 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-kube-api-access-2r629" pod "0d18657c-89fd-4f95-af19-096ef42f4e18" (UID: "0d18657c-89fd-4f95-af19-096ef42f4e18"). InnerVolumeSpecName "kube-api-access-2r629". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:48.150713 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.146643 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 30 17:24:48.150713 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.146719 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" podUID="25103151-7048-4796-adab-7e5b0b971e7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 30 17:24:48.154750 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.154708 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r629\" (UniqueName: \"kubernetes.io/projected/0d18657c-89fd-4f95-af19-096ef42f4e18-kube-api-access-2r629\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:24:48.446742 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.446697 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb" Oct 30 17:24:48.457918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.447373 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" Oct 30 17:24:48.457918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.447573 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7" event={"ID":"0d18657c-89fd-4f95-af19-096ef42f4e18","Type":"ContainerDied","Data":"078fc25286dc8fd91ba468da0e30977e4d9c2225d22854553f1c4a6220667389"} Oct 30 17:24:48.457918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.447651 4828 scope.go:122] "RemoveContainer" containerID="3542f6d908e1a4ffffae1d8a8b73c1532976292cb19fcae70583f4438485a602" Oct 30 17:24:48.467009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.466984 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 17:24:48.467115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.467027 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" podUID="44a0eb80-9692-4e67-a75b-43f638a8177c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 17:24:48.633181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.630859 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7"] Oct 30 17:24:48.639290 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:24:48.639242 4828 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c3138d_b28b_4d4f_9173_48d05241873f.slice/crio-0c62f09cd76a3183ff340c40e4b193692fb15d2a74326d47cc5d69329891aed4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c3138d_b28b_4d4f_9173_48d05241873f.slice\": RecentStats: unable to find data in memory cache]" Oct 30 17:24:48.640051 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.640022 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2d01ec-openshift-default-5b6654475c-b8gt7"] Oct 30 17:24:48.649859 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.649828 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb"] Oct 30 17:24:48.655542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:48.655506 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2d01ec-79b9ff8c9f-5fvbb"] Oct 30 17:24:49.318644 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:49.318609 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:49.319617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:49.319591 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:49.320758 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:49.320722 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:24:49.491926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:49.491458 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:49.495924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:49.492927 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:49.498759 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:49.497836 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:24:49.616975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:49.601200 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:50.494737 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:50.494705 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" path="/var/lib/kubelet/pods/0d18657c-89fd-4f95-af19-096ef42f4e18/volumes" Oct 30 17:24:50.495379 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:50.495358 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c3138d-b28b-4d4f-9173-48d05241873f" path="/var/lib/kubelet/pods/55c3138d-b28b-4d4f-9173-48d05241873f/volumes" Oct 30 17:24:51.203928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:51.196257 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:52.816166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:52.816126 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks"] Oct 30 17:24:52.879495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:52.879457 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:53.099411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:53.099276 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:54.050561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:54.050530 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt"] Oct 30 17:24:54.242142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:54.242106 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:54.405124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:54.405035 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:54.850781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:54.850746 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:56.514596 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:56.514558 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:58.528665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:58.528565 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:59.007997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:59.006451 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:24:59.068927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:59.063169 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" podUID="6fa7d177-2a63-46de-beda-a00ca0f153a4" containerName="payload-processing" containerID="cri-o://43412990b0d51f3c8341e8ec4b434e76ff2ecbc007599ddb0423db53ec9f06bf" gracePeriod=25 Oct 30 17:24:59.333413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:24:59.333322 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:00.816047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:00.816012 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:01.521767 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.521723 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps"] Oct 30 17:25:01.523263 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.523232 4828 cpu_manager.go:414] "containerMap: removing container" podUID="55c3138d-b28b-4d4f-9173-48d05241873f" containerName="payload-pre-processing" Oct 30 17:25:01.523523 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.523504 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c3138d-b28b-4d4f-9173-48d05241873f" containerName="payload-pre-processing" Oct 30 17:25:01.523677 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.523662 4828 cpu_manager.go:414] "containerMap: removing container" podUID="b1a87552-a6e2-451c-bd7c-e8b781c48aa4" containerName="main" Oct 30 17:25:01.523772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.523759 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a87552-a6e2-451c-bd7c-e8b781c48aa4" containerName="main" Oct 30 17:25:01.523949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.523934 4828 cpu_manager.go:414] "containerMap: removing container" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" Oct 30 17:25:01.524086 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.524053 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" Oct 30 17:25:01.524465 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.524431 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" Oct 30 17:25:01.524607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.524592 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c3138d-b28b-4d4f-9173-48d05241873f" containerName="payload-pre-processing" Oct 30 17:25:01.524711 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.524700 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a87552-a6e2-451c-bd7c-e8b781c48aa4" containerName="main" Oct 30 17:25:01.542722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.542614 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps"] Oct 30 17:25:01.542968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.542758 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.562178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.562144 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-model-cache\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.562321 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.562225 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-home\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.562404 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.562385 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c0ea1d5-82de-42de-a926-4c667fe4a066-tls-certs\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.562459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.562429 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-dshm\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.562508 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.562478 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz47n\" (UniqueName: \"kubernetes.io/projected/3c0ea1d5-82de-42de-a926-4c667fe4a066-kube-api-access-vz47n\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.562580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.562564 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-tmp-dir\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.605976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.594284 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-5d8947\"/\"default-dockercfg-gpj5b\"" Oct 30 17:25:01.605976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.594403 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-5d8947\"/\"openshift-service-ca.crt\"" Oct 30 17:25:01.605976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.594487 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-5d8947\"/\"kube-root-ca.crt\"" Oct 30 17:25:01.605976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.594624 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-5d8947\"/\"auth-test-model-5d8947-kserve-self-signed-certs\"" Oct 30 17:25:01.609955 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.609806 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:01.664278 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.664243 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c0ea1d5-82de-42de-a926-4c667fe4a066-tls-certs\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.664479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.664288 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-dshm\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.664479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.664334 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz47n\" (UniqueName: \"kubernetes.io/projected/3c0ea1d5-82de-42de-a926-4c667fe4a066-kube-api-access-vz47n\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.664479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.664391 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-tmp-dir\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.664704 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.664493 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-model-cache\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.664704 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.664533 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-home\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.664969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.664942 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-model-cache\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.665064 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.665022 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-tmp-dir\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.665064 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.665046 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-home\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.666776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.666747 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-dshm\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.667239 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.667217 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c0ea1d5-82de-42de-a926-4c667fe4a066-tls-certs\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.735920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.735728 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz47n\" (UniqueName: \"kubernetes.io/projected/3c0ea1d5-82de-42de-a926-4c667fe4a066-kube-api-access-vz47n\") pod \"auth-test-model-5d8947-kserve-5666f7c8df-fbnps\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:01.870982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:01.870863 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:02.268193 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:25:02.268160 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c0ea1d5_82de_42de_a926_4c667fe4a066.slice/crio-0476c6ab07d4cf3239edd3b7ce08ba40cf46d228b4ab889854b1b011b4ddbe79 WatchSource:0}: Error finding container 0476c6ab07d4cf3239edd3b7ce08ba40cf46d228b4ab889854b1b011b4ddbe79: Status 404 returned error can't find the container with id 0476c6ab07d4cf3239edd3b7ce08ba40cf46d228b4ab889854b1b011b4ddbe79 Oct 30 17:25:02.278923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:02.269030 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps"] Oct 30 17:25:02.756311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:02.756202 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:02.785995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:02.785956 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" event={"ID":"3c0ea1d5-82de-42de-a926-4c667fe4a066","Type":"ContainerStarted","Data":"00e02e86526b4bad934348cec54c97133e49d198fd035557c0ef4e4c638f7b53"} Oct 30 17:25:02.786230 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:02.786210 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" event={"ID":"3c0ea1d5-82de-42de-a926-4c667fe4a066","Type":"ContainerStarted","Data":"0476c6ab07d4cf3239edd3b7ce08ba40cf46d228b4ab889854b1b011b4ddbe79"} Oct 30 17:25:02.834184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:02.834124 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" podStartSLOduration=1.834109059 podStartE2EDuration="1.834109059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:02.830233025 +0000 UTC m=+2280.475743205" watchObservedRunningTime="2026-10-30 17:25:02.834109059 +0000 UTC m=+2280.479619262" Oct 30 17:25:02.881098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:02.881052 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:03.603455 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:03.603412 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" podUID="6fa7d177-2a63-46de-beda-a00ca0f153a4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:04.047910 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:04.047859 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:04.811924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:04.800605 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:06.600546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:06.600469 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:07.840353 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:07.840295 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" podUID="a5e47918-6c04-453d-b6ed-fabc181b3807" containerName="main" containerID="cri-o://293a667a36317a929a86227ff25a002ee54562421317d060244df9c9bfee226f" gracePeriod=45 Oct 30 17:25:07.916501 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:25:07.916457 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:25:08.183030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.174002 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:08.230023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.229988 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:25:08.318223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.318143 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-home" pod "a5e47918-6c04-453d-b6ed-fabc181b3807" (UID: "a5e47918-6c04-453d-b6ed-fabc181b3807"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:08.320686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.318270 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-home\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-home\") pod \"a5e47918-6c04-453d-b6ed-fabc181b3807\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " Oct 30 17:25:08.320686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.318354 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e47918-6c04-453d-b6ed-fabc181b3807-kube-api-access-5mvhs\" (UniqueName: \"kubernetes.io/projected/a5e47918-6c04-453d-b6ed-fabc181b3807-kube-api-access-5mvhs\") pod \"a5e47918-6c04-453d-b6ed-fabc181b3807\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " Oct 30 17:25:08.320686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.318465 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-model-cache\") pod \"a5e47918-6c04-453d-b6ed-fabc181b3807\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " Oct 30 17:25:08.320686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.318782 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-model-cache" pod "a5e47918-6c04-453d-b6ed-fabc181b3807" (UID: "a5e47918-6c04-453d-b6ed-fabc181b3807"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:08.320686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.318534 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-tmp-dir\") pod \"a5e47918-6c04-453d-b6ed-fabc181b3807\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " Oct 30 17:25:08.320686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.318999 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-dshm\") pod \"a5e47918-6c04-453d-b6ed-fabc181b3807\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " Oct 30 17:25:08.320686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.319371 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e47918-6c04-453d-b6ed-fabc181b3807-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e47918-6c04-453d-b6ed-fabc181b3807-tls-certs\") pod \"a5e47918-6c04-453d-b6ed-fabc181b3807\" (UID: \"a5e47918-6c04-453d-b6ed-fabc181b3807\") " Oct 30 17:25:08.320686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.320137 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:08.320686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.320157 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:08.320686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.319033 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-tmp-dir" pod "a5e47918-6c04-453d-b6ed-fabc181b3807" (UID: "a5e47918-6c04-453d-b6ed-fabc181b3807"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:08.321430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.321401 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e47918-6c04-453d-b6ed-fabc181b3807-kube-api-access-5mvhs" pod "a5e47918-6c04-453d-b6ed-fabc181b3807" (UID: "a5e47918-6c04-453d-b6ed-fabc181b3807"). InnerVolumeSpecName "kube-api-access-5mvhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:08.321860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.321802 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-dshm" pod "a5e47918-6c04-453d-b6ed-fabc181b3807" (UID: "a5e47918-6c04-453d-b6ed-fabc181b3807"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:08.322894 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.322868 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e47918-6c04-453d-b6ed-fabc181b3807-tls-certs" pod "a5e47918-6c04-453d-b6ed-fabc181b3807" (UID: "a5e47918-6c04-453d-b6ed-fabc181b3807"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:08.421400 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.421366 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mvhs\" (UniqueName: \"kubernetes.io/projected/a5e47918-6c04-453d-b6ed-fabc181b3807-kube-api-access-5mvhs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:08.421586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.421418 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:08.421586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.421436 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5e47918-6c04-453d-b6ed-fabc181b3807-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:08.421586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.421449 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5e47918-6c04-453d-b6ed-fabc181b3807-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:08.886499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.886466 4828 generic.go:356] "Generic (PLEG): container finished" podID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerID="e2a2a6d091bbc13b550b409f779e3e3234b92a3827254279450b685a2e618cd0" exitCode=0 Oct 30 17:25:08.887035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.886540 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" event={"ID":"5f3f407f-bf79-4e39-8c17-c1e2a33eadae","Type":"ContainerDied","Data":"e2a2a6d091bbc13b550b409f779e3e3234b92a3827254279450b685a2e618cd0"} Oct 30 17:25:08.887159 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.887137 4828 scope.go:122] "RemoveContainer" containerID="e2a2a6d091bbc13b550b409f779e3e3234b92a3827254279450b685a2e618cd0" Oct 30 17:25:08.888744 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.888719 4828 generic.go:356] "Generic (PLEG): container finished" podID="a5e47918-6c04-453d-b6ed-fabc181b3807" containerID="293a667a36317a929a86227ff25a002ee54562421317d060244df9c9bfee226f" exitCode=0 Oct 30 17:25:08.889033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.889012 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" event={"ID":"a5e47918-6c04-453d-b6ed-fabc181b3807","Type":"ContainerDied","Data":"293a667a36317a929a86227ff25a002ee54562421317d060244df9c9bfee226f"} Oct 30 17:25:08.889132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.889051 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" event={"ID":"a5e47918-6c04-453d-b6ed-fabc181b3807","Type":"ContainerDied","Data":"51d7a389f207bee00c8642fb254b20451e54cdecc9553feea8f2e8a398290707"} Oct 30 17:25:08.889132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.889072 4828 scope.go:122] "RemoveContainer" containerID="293a667a36317a929a86227ff25a002ee54562421317d060244df9c9bfee226f" Oct 30 17:25:08.889246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.889156 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks" Oct 30 17:25:08.889653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.889626 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:25:08.889733 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.889703 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:25:08.889798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.889757 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:25:08.914683 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.914652 4828 scope.go:122] "RemoveContainer" containerID="293a667a36317a929a86227ff25a002ee54562421317d060244df9c9bfee226f" Oct 30 17:25:08.915090 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:25:08.915052 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293a667a36317a929a86227ff25a002ee54562421317d060244df9c9bfee226f\": container with ID starting with 293a667a36317a929a86227ff25a002ee54562421317d060244df9c9bfee226f not found: ID does not exist" containerID="293a667a36317a929a86227ff25a002ee54562421317d060244df9c9bfee226f" Oct 30 17:25:08.915189 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.915107 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293a667a36317a929a86227ff25a002ee54562421317d060244df9c9bfee226f"} err="failed to get container status \"293a667a36317a929a86227ff25a002ee54562421317d060244df9c9bfee226f\": rpc error: code = NotFound desc = could not find container \"293a667a36317a929a86227ff25a002ee54562421317d060244df9c9bfee226f\": container with ID starting with 293a667a36317a929a86227ff25a002ee54562421317d060244df9c9bfee226f not found: ID does not exist" Oct 30 17:25:08.991972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.990416 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks"] Oct 30 17:25:08.999207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:08.998702 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-987dfc/auth-test-model-987dfc-kserve-5479d79d85-sh2ks"] Oct 30 17:25:09.926247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:09.924444 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" event={"ID":"5f3f407f-bf79-4e39-8c17-c1e2a33eadae","Type":"ContainerStarted","Data":"39d53b47ea2b78af9e57bd31ca3d1e2da25bf9ebb0a05ee56716d908d40a2a5e"} Oct 30 17:25:10.043023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:10.042982 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:25:10.043239 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:10.043210 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 17:25:10.043318 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:10.043266 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 17:25:10.068426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:10.052301 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:10.509920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:10.509870 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e47918-6c04-453d-b6ed-fabc181b3807" path="/var/lib/kubelet/pods/a5e47918-6c04-453d-b6ed-fabc181b3807/volumes" Oct 30 17:25:11.043451 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:11.043418 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 17:25:11.043862 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:11.043472 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 17:25:11.344715 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:11.344619 4828 scope.go:122] "RemoveContainer" containerID="609a7fe64677fa57d3787e20b3e6f99785bbae7b05c3cf4f772aaf6550957287" Oct 30 17:25:11.773925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:11.769437 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:11.886925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:11.873333 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:11.918984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:11.918957 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:11.954664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:11.954635 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:25:12.043147 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:12.043059 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 17:25:12.043297 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:12.043122 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 17:25:13.042889 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:13.042860 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 17:25:13.043397 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:13.042927 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 17:25:13.464484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:13.463636 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:13.616992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:13.616573 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" podUID="6fa7d177-2a63-46de-beda-a00ca0f153a4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:14.043501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:14.043459 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 17:25:14.043967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:14.043520 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 17:25:14.254423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:14.254313 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd"] Oct 30 17:25:14.457393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:14.457299 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:14.647922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:14.646882 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:15.038778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.038271 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:15.055925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.055354 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 17:25:15.055925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.055441 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 17:25:15.425981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.424699 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:15.770948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.770863 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:15.800059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.800002 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf"] Oct 30 17:25:15.803921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.801265 4828 cpu_manager.go:414] "containerMap: removing container" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" Oct 30 17:25:15.803921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.801290 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" Oct 30 17:25:15.803921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.801325 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a5e47918-6c04-453d-b6ed-fabc181b3807" containerName="main" Oct 30 17:25:15.803921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.801335 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e47918-6c04-453d-b6ed-fabc181b3807" containerName="main" Oct 30 17:25:15.803921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.801517 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d18657c-89fd-4f95-af19-096ef42f4e18" containerName="istio-proxy" Oct 30 17:25:15.803921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.801528 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e47918-6c04-453d-b6ed-fabc181b3807" containerName="main" Oct 30 17:25:15.810035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.808363 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:15.835213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.835182 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-abe243-openshift-default-dockercfg-bj4v9\"" Oct 30 17:25:15.837522 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.837488 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf"] Oct 30 17:25:15.923717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.923667 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7hxx\" (UniqueName: \"kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-kube-api-access-g7hxx\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:15.923894 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.923786 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-data\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:15.923894 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.923837 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-token\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:15.923894 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.923923 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istiod-ca-cert\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:15.924055 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.924014 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-socket\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:15.924135 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.924114 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-envoy\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:15.924209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.924197 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-podinfo\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:15.924288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.924261 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-certs\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:15.924331 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.924312 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-credential-socket\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:15.925647 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:15.925615 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:16.033247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.033140 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-envoy\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.033572 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.033549 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-podinfo\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.033719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.033703 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-certs\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.034605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.034580 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-credential-socket\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.035666 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.035642 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7hxx\" (UniqueName: \"kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-kube-api-access-g7hxx\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.036383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.036361 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-data\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.036578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.036558 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-token\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.036694 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.036680 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istiod-ca-cert\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.036830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.036814 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-socket\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.046932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.037331 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-socket\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.049149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.034348 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-certs\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.049359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.035399 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-credential-socket\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.050069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.047565 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-data\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.050255 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.048614 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istiod-ca-cert\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.099923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.087020 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-podinfo\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.099923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.091519 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-envoy\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.103074 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.101617 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-token\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.134922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.131286 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7hxx\" (UniqueName: \"kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-kube-api-access-g7hxx\") pod \"e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.156941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.142524 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:16.165638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.165478 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:16.543344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.543308 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:25:16.574996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.574957 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:25:16.640215 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.640177 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:25:16.640332 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.640249 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:25:16.640332 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.640283 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:25:16.641116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:16.641093 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf"] Oct 30 17:25:17.139976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:17.139935 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" event={"ID":"7e0b2768-4823-483d-91d6-b4bd5df2b78e","Type":"ContainerStarted","Data":"d7f5c6fdc49ca2b5a7b221cfa8a101002f6e42e9b840b073d7c7129932959155"} Oct 30 17:25:17.140407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:17.139998 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" event={"ID":"7e0b2768-4823-483d-91d6-b4bd5df2b78e","Type":"ContainerStarted","Data":"b486ed95716c8734749419fcd66c25076c1a0a50f336a7ef13dd82bea8541138"} Oct 30 17:25:17.141164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:17.141147 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:25:17.157513 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:17.157489 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:17.164047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:17.164027 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:17.189183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:17.189138 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" podStartSLOduration=2.189125077 podStartE2EDuration="2.189125077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:17.186790344 +0000 UTC m=+2294.832300534" watchObservedRunningTime="2026-10-30 17:25:17.189125077 +0000 UTC m=+2294.834635257" Oct 30 17:25:17.622829 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:17.622789 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:18.148954 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:18.148859 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:18.150280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:18.150252 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:25:19.082796 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:19.082616 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:19.256662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:19.256553 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" podUID="7eb5f6d5-47f2-4516-9a67-769660921f0c" containerName="payload-processing" containerID="cri-o://3b5bc05ccc86264d72b56e0802380e4781ce0efb3a801178fc93c9f4b364dc57" gracePeriod=25 Oct 30 17:25:19.797923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:19.789723 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:19.889119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:19.886357 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" Oct 30 17:25:19.919437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:19.918562 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa7d177-2a63-46de-beda-a00ca0f153a4-kube-api-access-j8tvf\" (UniqueName: \"kubernetes.io/projected/6fa7d177-2a63-46de-beda-a00ca0f153a4-kube-api-access-j8tvf\") pod \"6fa7d177-2a63-46de-beda-a00ca0f153a4\" (UID: \"6fa7d177-2a63-46de-beda-a00ca0f153a4\") " Oct 30 17:25:19.919437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:19.918818 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa7d177-2a63-46de-beda-a00ca0f153a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa7d177-2a63-46de-beda-a00ca0f153a4-plugins-config-volume\") pod \"6fa7d177-2a63-46de-beda-a00ca0f153a4\" (UID: \"6fa7d177-2a63-46de-beda-a00ca0f153a4\") " Oct 30 17:25:19.919437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:19.919339 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa7d177-2a63-46de-beda-a00ca0f153a4-plugins-config-volume" pod "6fa7d177-2a63-46de-beda-a00ca0f153a4" (UID: "6fa7d177-2a63-46de-beda-a00ca0f153a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:19.920005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:19.919969 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa7d177-2a63-46de-beda-a00ca0f153a4-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:19.921361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:19.921340 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa7d177-2a63-46de-beda-a00ca0f153a4-kube-api-access-j8tvf" pod "6fa7d177-2a63-46de-beda-a00ca0f153a4" (UID: "6fa7d177-2a63-46de-beda-a00ca0f153a4"). InnerVolumeSpecName "kube-api-access-j8tvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:19.973357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:19.973332 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" Oct 30 17:25:19.999722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:19.999597 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:20.021460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.021429 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eb5f6d5-47f2-4516-9a67-769660921f0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eb5f6d5-47f2-4516-9a67-769660921f0c-plugins-config-volume\") pod \"7eb5f6d5-47f2-4516-9a67-769660921f0c\" (UID: \"7eb5f6d5-47f2-4516-9a67-769660921f0c\") " Oct 30 17:25:20.021661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.021534 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb5f6d5-47f2-4516-9a67-769660921f0c-kube-api-access-bzh5n\" (UniqueName: \"kubernetes.io/projected/7eb5f6d5-47f2-4516-9a67-769660921f0c-kube-api-access-bzh5n\") pod \"7eb5f6d5-47f2-4516-9a67-769660921f0c\" (UID: \"7eb5f6d5-47f2-4516-9a67-769660921f0c\") " Oct 30 17:25:20.022423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.022320 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb5f6d5-47f2-4516-9a67-769660921f0c-plugins-config-volume" pod "7eb5f6d5-47f2-4516-9a67-769660921f0c" (UID: "7eb5f6d5-47f2-4516-9a67-769660921f0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:20.022561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.022479 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eb5f6d5-47f2-4516-9a67-769660921f0c-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:20.022561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.022503 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8tvf\" (UniqueName: \"kubernetes.io/projected/6fa7d177-2a63-46de-beda-a00ca0f153a4-kube-api-access-j8tvf\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:20.024280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.024255 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb5f6d5-47f2-4516-9a67-769660921f0c-kube-api-access-bzh5n" pod "7eb5f6d5-47f2-4516-9a67-769660921f0c" (UID: "7eb5f6d5-47f2-4516-9a67-769660921f0c"). InnerVolumeSpecName "kube-api-access-bzh5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:20.124618 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.124525 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzh5n\" (UniqueName: \"kubernetes.io/projected/7eb5f6d5-47f2-4516-9a67-769660921f0c-kube-api-access-bzh5n\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:20.148479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.148441 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:20.178648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.178610 4828 generic.go:356] "Generic (PLEG): container finished" podID="7eb5f6d5-47f2-4516-9a67-769660921f0c" containerID="3b5bc05ccc86264d72b56e0802380e4781ce0efb3a801178fc93c9f4b364dc57" exitCode=0 Oct 30 17:25:20.178853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.178690 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" Oct 30 17:25:20.178853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.178701 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" event={"ID":"7eb5f6d5-47f2-4516-9a67-769660921f0c","Type":"ContainerDied","Data":"3b5bc05ccc86264d72b56e0802380e4781ce0efb3a801178fc93c9f4b364dc57"} Oct 30 17:25:20.178853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.178786 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd" event={"ID":"7eb5f6d5-47f2-4516-9a67-769660921f0c","Type":"ContainerDied","Data":"be7ff0c89b0a2cd0af9cbc0272ea0e7918cb92a394800e209567a5a132b89fcb"} Oct 30 17:25:20.178853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.178817 4828 scope.go:122] "RemoveContainer" containerID="3b5bc05ccc86264d72b56e0802380e4781ce0efb3a801178fc93c9f4b364dc57" Oct 30 17:25:20.186649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.186395 4828 generic.go:356] "Generic (PLEG): container finished" podID="6fa7d177-2a63-46de-beda-a00ca0f153a4" containerID="43412990b0d51f3c8341e8ec4b434e76ff2ecbc007599ddb0423db53ec9f06bf" exitCode=0 Oct 30 17:25:20.186649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.186516 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" Oct 30 17:25:20.186649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.186513 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" event={"ID":"6fa7d177-2a63-46de-beda-a00ca0f153a4","Type":"ContainerDied","Data":"43412990b0d51f3c8341e8ec4b434e76ff2ecbc007599ddb0423db53ec9f06bf"} Oct 30 17:25:20.186977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.186647 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt" event={"ID":"6fa7d177-2a63-46de-beda-a00ca0f153a4","Type":"ContainerDied","Data":"2a36c10ad219b8633f2510efeec097cbb61aad3fdfab22305845f5ddf16fe30f"} Oct 30 17:25:20.193381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.193354 4828 scope.go:122] "RemoveContainer" containerID="3b5bc05ccc86264d72b56e0802380e4781ce0efb3a801178fc93c9f4b364dc57" Oct 30 17:25:20.193833 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:25:20.193798 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5bc05ccc86264d72b56e0802380e4781ce0efb3a801178fc93c9f4b364dc57\": container with ID starting with 3b5bc05ccc86264d72b56e0802380e4781ce0efb3a801178fc93c9f4b364dc57 not found: ID does not exist" containerID="3b5bc05ccc86264d72b56e0802380e4781ce0efb3a801178fc93c9f4b364dc57" Oct 30 17:25:20.194044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.193843 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5bc05ccc86264d72b56e0802380e4781ce0efb3a801178fc93c9f4b364dc57"} err="failed to get container status \"3b5bc05ccc86264d72b56e0802380e4781ce0efb3a801178fc93c9f4b364dc57\": rpc error: code = NotFound desc = could not find container \"3b5bc05ccc86264d72b56e0802380e4781ce0efb3a801178fc93c9f4b364dc57\": container with ID starting with 3b5bc05ccc86264d72b56e0802380e4781ce0efb3a801178fc93c9f4b364dc57 not found: ID does not exist" Oct 30 17:25:20.194044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.193868 4828 scope.go:122] "RemoveContainer" containerID="43412990b0d51f3c8341e8ec4b434e76ff2ecbc007599ddb0423db53ec9f06bf" Oct 30 17:25:20.238203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.238173 4828 scope.go:122] "RemoveContainer" containerID="43412990b0d51f3c8341e8ec4b434e76ff2ecbc007599ddb0423db53ec9f06bf" Oct 30 17:25:20.260921 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:25:20.248998 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43412990b0d51f3c8341e8ec4b434e76ff2ecbc007599ddb0423db53ec9f06bf\": container with ID starting with 43412990b0d51f3c8341e8ec4b434e76ff2ecbc007599ddb0423db53ec9f06bf not found: ID does not exist" containerID="43412990b0d51f3c8341e8ec4b434e76ff2ecbc007599ddb0423db53ec9f06bf" Oct 30 17:25:20.260921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.249050 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43412990b0d51f3c8341e8ec4b434e76ff2ecbc007599ddb0423db53ec9f06bf"} err="failed to get container status \"43412990b0d51f3c8341e8ec4b434e76ff2ecbc007599ddb0423db53ec9f06bf\": rpc error: code = NotFound desc = could not find container \"43412990b0d51f3c8341e8ec4b434e76ff2ecbc007599ddb0423db53ec9f06bf\": container with ID starting with 43412990b0d51f3c8341e8ec4b434e76ff2ecbc007599ddb0423db53ec9f06bf not found: ID does not exist" Oct 30 17:25:20.283735 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.283606 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd"] Oct 30 17:25:20.288419 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.288381 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-42d381-6997bdb576-xz5hd"] Oct 30 17:25:20.308953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.308867 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:20.309644 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.309615 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt"] Oct 30 17:25:20.314636 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.314606 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e5ac8f-6d7cf89854-p8bdt"] Oct 30 17:25:20.492577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.492492 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa7d177-2a63-46de-beda-a00ca0f153a4" path="/var/lib/kubelet/pods/6fa7d177-2a63-46de-beda-a00ca0f153a4/volumes" Oct 30 17:25:20.492999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.492983 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eb5f6d5-47f2-4516-9a67-769660921f0c" path="/var/lib/kubelet/pods/7eb5f6d5-47f2-4516-9a67-769660921f0c/volumes" Oct 30 17:25:20.708276 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:20.707629 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:21.591477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:21.591437 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:21.755713 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:21.755674 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:22.023925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.018984 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d"] Oct 30 17:25:22.038154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.037245 4828 cpu_manager.go:414] "containerMap: removing container" podUID="6fa7d177-2a63-46de-beda-a00ca0f153a4" containerName="payload-processing" Oct 30 17:25:22.038154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.037283 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa7d177-2a63-46de-beda-a00ca0f153a4" containerName="payload-processing" Oct 30 17:25:22.038154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.037303 4828 cpu_manager.go:414] "containerMap: removing container" podUID="7eb5f6d5-47f2-4516-9a67-769660921f0c" containerName="payload-processing" Oct 30 17:25:22.038154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.037324 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb5f6d5-47f2-4516-9a67-769660921f0c" containerName="payload-processing" Oct 30 17:25:22.038154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.037639 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa7d177-2a63-46de-beda-a00ca0f153a4" containerName="payload-processing" Oct 30 17:25:22.038154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.037659 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb5f6d5-47f2-4516-9a67-769660921f0c" containerName="payload-processing" Oct 30 17:25:22.111505 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.111232 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d"] Oct 30 17:25:22.111505 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.111393 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" Oct 30 17:25:22.137415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.136209 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-abe243\"" Oct 30 17:25:22.143962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.141693 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:22.149080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.149038 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-abe243-dockercfg-ztpcd\"" Oct 30 17:25:22.322288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.322195 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6024600-42ca-45d6-97e5-cc0ae8071f3b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d\" (UID: \"b6024600-42ca-45d6-97e5-cc0ae8071f3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" Oct 30 17:25:22.322453 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.322343 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sggf9\" (UniqueName: \"kubernetes.io/projected/b6024600-42ca-45d6-97e5-cc0ae8071f3b-kube-api-access-sggf9\") pod \"payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d\" (UID: \"b6024600-42ca-45d6-97e5-cc0ae8071f3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" Oct 30 17:25:22.423435 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.423401 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6024600-42ca-45d6-97e5-cc0ae8071f3b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d\" (UID: \"b6024600-42ca-45d6-97e5-cc0ae8071f3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" Oct 30 17:25:22.423616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.423464 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sggf9\" (UniqueName: \"kubernetes.io/projected/b6024600-42ca-45d6-97e5-cc0ae8071f3b-kube-api-access-sggf9\") pod \"payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d\" (UID: \"b6024600-42ca-45d6-97e5-cc0ae8071f3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" Oct 30 17:25:22.424270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.424243 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6024600-42ca-45d6-97e5-cc0ae8071f3b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d\" (UID: \"b6024600-42ca-45d6-97e5-cc0ae8071f3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" Oct 30 17:25:22.448353 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.448316 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sggf9\" (UniqueName: \"kubernetes.io/projected/b6024600-42ca-45d6-97e5-cc0ae8071f3b-kube-api-access-sggf9\") pod \"payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d\" (UID: \"b6024600-42ca-45d6-97e5-cc0ae8071f3b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" Oct 30 17:25:22.738850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:22.738760 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" Oct 30 17:25:23.052168 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:25:23.052138 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6024600_42ca_45d6_97e5_cc0ae8071f3b.slice/crio-3df6b1f7942ec71eac370bbbe06292e851fc34ad6c64fdb8ce133f81c182e999 WatchSource:0}: Error finding container 3df6b1f7942ec71eac370bbbe06292e851fc34ad6c64fdb8ce133f81c182e999: Status 404 returned error can't find the container with id 3df6b1f7942ec71eac370bbbe06292e851fc34ad6c64fdb8ce133f81c182e999 Oct 30 17:25:23.054892 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:23.054862 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d"] Oct 30 17:25:23.274682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:23.274644 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" event={"ID":"b6024600-42ca-45d6-97e5-cc0ae8071f3b","Type":"ContainerStarted","Data":"b4d4020998dc13e277a8293003ac41f92291711ba7086bf4a3c82191d2872949"} Oct 30 17:25:23.274881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:23.274730 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" event={"ID":"b6024600-42ca-45d6-97e5-cc0ae8071f3b","Type":"ContainerStarted","Data":"3df6b1f7942ec71eac370bbbe06292e851fc34ad6c64fdb8ce133f81c182e999"} Oct 30 17:25:23.274881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:23.274756 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" Oct 30 17:25:23.347946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:23.344914 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" podStartSLOduration=2.34487783 podStartE2EDuration="2.34487783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:23.335072467 +0000 UTC m=+2300.980582646" watchObservedRunningTime="2026-10-30 17:25:23.34487783 +0000 UTC m=+2300.990388011" Oct 30 17:25:23.907326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:23.907283 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:25.527450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:25.527406 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:27.352768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:27.352728 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:27.502303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:27.502265 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:27.822565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:27.822529 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk"] Oct 30 17:25:27.822799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:27.822770 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" containerID="cri-o://26d5a9c7ebd6a4d967136327a9842a37d7db93bab9635b9eb876442ad6792c47" gracePeriod=30 Oct 30 17:25:27.872166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:27.872128 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:28.713946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:28.713913 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:29.232509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:29.232466 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:29.399481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:29.399446 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:31.043351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:31.043312 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:32.679193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:32.679151 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:32.853113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:32.853066 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:33.013718 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.013677 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:33.323701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.323673 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:25:33.410362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.410327 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-credential-socket\") pod \"1882faef-b475-4456-888e-4ecb07d50831\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " Oct 30 17:25:33.410571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.410401 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-istio-token\" (UniqueName: \"kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-istio-token\") pod \"1882faef-b475-4456-888e-4ecb07d50831\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " Oct 30 17:25:33.410571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.410463 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-kube-api-access-wtz4l\" (UniqueName: \"kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-kube-api-access-wtz4l\") pod \"1882faef-b475-4456-888e-4ecb07d50831\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " Oct 30 17:25:33.410571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.410549 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1882faef-b475-4456-888e-4ecb07d50831-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1882faef-b475-4456-888e-4ecb07d50831-istio-podinfo\") pod \"1882faef-b475-4456-888e-4ecb07d50831\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " Oct 30 17:25:33.410751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.410637 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-data\") pod \"1882faef-b475-4456-888e-4ecb07d50831\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " Oct 30 17:25:33.410751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.410644 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-credential-socket" pod "1882faef-b475-4456-888e-4ecb07d50831" (UID: "1882faef-b475-4456-888e-4ecb07d50831"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:33.410751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.410692 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-envoy\") pod \"1882faef-b475-4456-888e-4ecb07d50831\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " Oct 30 17:25:33.410751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.410724 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1882faef-b475-4456-888e-4ecb07d50831-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1882faef-b475-4456-888e-4ecb07d50831-istiod-ca-cert\") pod \"1882faef-b475-4456-888e-4ecb07d50831\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " Oct 30 17:25:33.410983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.410755 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-socket\") pod \"1882faef-b475-4456-888e-4ecb07d50831\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " Oct 30 17:25:33.410983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.410843 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-certs\") pod \"1882faef-b475-4456-888e-4ecb07d50831\" (UID: \"1882faef-b475-4456-888e-4ecb07d50831\") " Oct 30 17:25:33.411540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.411513 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-data" pod "1882faef-b475-4456-888e-4ecb07d50831" (UID: "1882faef-b475-4456-888e-4ecb07d50831"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:33.411744 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.411539 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:33.411882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.411777 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-certs" pod "1882faef-b475-4456-888e-4ecb07d50831" (UID: "1882faef-b475-4456-888e-4ecb07d50831"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:33.413536 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.413502 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-envoy" pod "1882faef-b475-4456-888e-4ecb07d50831" (UID: "1882faef-b475-4456-888e-4ecb07d50831"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:33.418583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.418552 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-kube-api-access-wtz4l" pod "1882faef-b475-4456-888e-4ecb07d50831" (UID: "1882faef-b475-4456-888e-4ecb07d50831"). InnerVolumeSpecName "kube-api-access-wtz4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:33.418790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.418766 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1882faef-b475-4456-888e-4ecb07d50831-istio-podinfo" pod "1882faef-b475-4456-888e-4ecb07d50831" (UID: "1882faef-b475-4456-888e-4ecb07d50831"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:25:33.419999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.419974 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-socket" pod "1882faef-b475-4456-888e-4ecb07d50831" (UID: "1882faef-b475-4456-888e-4ecb07d50831"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:33.420134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.420047 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1882faef-b475-4456-888e-4ecb07d50831-istiod-ca-cert" pod "1882faef-b475-4456-888e-4ecb07d50831" (UID: "1882faef-b475-4456-888e-4ecb07d50831"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:33.421724 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.421697 4828 generic.go:356] "Generic (PLEG): container finished" podID="1882faef-b475-4456-888e-4ecb07d50831" containerID="26d5a9c7ebd6a4d967136327a9842a37d7db93bab9635b9eb876442ad6792c47" exitCode=0 Oct 30 17:25:33.422039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.422018 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" event={"ID":"1882faef-b475-4456-888e-4ecb07d50831","Type":"ContainerDied","Data":"26d5a9c7ebd6a4d967136327a9842a37d7db93bab9635b9eb876442ad6792c47"} Oct 30 17:25:33.422133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.422068 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" event={"ID":"1882faef-b475-4456-888e-4ecb07d50831","Type":"ContainerDied","Data":"6b656e8fe13d3d02b9354fc28413e3419ae747cb25d6fd452ffefa4feb5838b8"} Oct 30 17:25:33.422133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.422091 4828 scope.go:122] "RemoveContainer" containerID="26d5a9c7ebd6a4d967136327a9842a37d7db93bab9635b9eb876442ad6792c47" Oct 30 17:25:33.422250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.422227 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk" Oct 30 17:25:33.422385 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.422361 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-istio-token" pod "1882faef-b475-4456-888e-4ecb07d50831" (UID: "1882faef-b475-4456-888e-4ecb07d50831"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:33.453225 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.453170 4828 scope.go:122] "RemoveContainer" containerID="97e63113f26c350f1a587a9fde05592303709113bc4c69b451391a463a6d4970" Oct 30 17:25:33.467071 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.467047 4828 scope.go:122] "RemoveContainer" containerID="26d5a9c7ebd6a4d967136327a9842a37d7db93bab9635b9eb876442ad6792c47" Oct 30 17:25:33.467438 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:25:33.467414 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d5a9c7ebd6a4d967136327a9842a37d7db93bab9635b9eb876442ad6792c47\": container with ID starting with 26d5a9c7ebd6a4d967136327a9842a37d7db93bab9635b9eb876442ad6792c47 not found: ID does not exist" containerID="26d5a9c7ebd6a4d967136327a9842a37d7db93bab9635b9eb876442ad6792c47" Oct 30 17:25:33.467493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.467449 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d5a9c7ebd6a4d967136327a9842a37d7db93bab9635b9eb876442ad6792c47"} err="failed to get container status \"26d5a9c7ebd6a4d967136327a9842a37d7db93bab9635b9eb876442ad6792c47\": rpc error: code = NotFound desc = could not find container \"26d5a9c7ebd6a4d967136327a9842a37d7db93bab9635b9eb876442ad6792c47\": container with ID starting with 26d5a9c7ebd6a4d967136327a9842a37d7db93bab9635b9eb876442ad6792c47 not found: ID does not exist" Oct 30 17:25:33.467493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.467469 4828 scope.go:122] "RemoveContainer" containerID="97e63113f26c350f1a587a9fde05592303709113bc4c69b451391a463a6d4970" Oct 30 17:25:33.467763 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:25:33.467742 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e63113f26c350f1a587a9fde05592303709113bc4c69b451391a463a6d4970\": container with ID starting with 97e63113f26c350f1a587a9fde05592303709113bc4c69b451391a463a6d4970 not found: ID does not exist" containerID="97e63113f26c350f1a587a9fde05592303709113bc4c69b451391a463a6d4970" Oct 30 17:25:33.467806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.467770 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e63113f26c350f1a587a9fde05592303709113bc4c69b451391a463a6d4970"} err="failed to get container status \"97e63113f26c350f1a587a9fde05592303709113bc4c69b451391a463a6d4970\": rpc error: code = NotFound desc = could not find container \"97e63113f26c350f1a587a9fde05592303709113bc4c69b451391a463a6d4970\": container with ID starting with 97e63113f26c350f1a587a9fde05592303709113bc4c69b451391a463a6d4970 not found: ID does not exist" Oct 30 17:25:33.513013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.512961 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:33.513013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.512992 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1882faef-b475-4456-888e-4ecb07d50831-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:33.513013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.513001 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:33.513013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.513010 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:33.513013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.513019 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:33.513306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.513028 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtz4l\" (UniqueName: \"kubernetes.io/projected/1882faef-b475-4456-888e-4ecb07d50831-kube-api-access-wtz4l\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:33.513306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.513039 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1882faef-b475-4456-888e-4ecb07d50831-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:33.513306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.513049 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1882faef-b475-4456-888e-4ecb07d50831-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:33.775147 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.775105 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk"] Oct 30 17:25:33.788150 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:33.788116 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e5ac8f-openshift-default-65c475fcc-lchfk"] Oct 30 17:25:34.287562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.287529 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" Oct 30 17:25:34.366925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.365961 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:34.493594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.493559 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1882faef-b475-4456-888e-4ecb07d50831" path="/var/lib/kubelet/pods/1882faef-b475-4456-888e-4ecb07d50831/volumes" Oct 30 17:25:34.494701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.494676 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75"] Oct 30 17:25:34.495659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.495634 4828 cpu_manager.go:414] "containerMap: removing container" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" Oct 30 17:25:34.495659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.495656 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" Oct 30 17:25:34.495806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.495678 4828 cpu_manager.go:414] "containerMap: removing container" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" Oct 30 17:25:34.495806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.495684 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" Oct 30 17:25:34.495806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.495801 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" Oct 30 17:25:34.495895 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.495810 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="1882faef-b475-4456-888e-4ecb07d50831" containerName="istio-proxy" Oct 30 17:25:34.501099 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.501081 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" Oct 30 17:25:34.520802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.520773 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-469ad7\"" Oct 30 17:25:34.521002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.520986 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-469ad7-dockercfg-g94jj\"" Oct 30 17:25:34.521946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.521921 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75"] Oct 30 17:25:34.524780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.524731 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:34.527167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.527112 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6cfq\" (UniqueName: \"kubernetes.io/projected/1b3a84af-dd51-40b7-b426-b51d208ca058-kube-api-access-n6cfq\") pod \"payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75\" (UID: \"1b3a84af-dd51-40b7-b426-b51d208ca058\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" Oct 30 17:25:34.527488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.527466 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3a84af-dd51-40b7-b426-b51d208ca058-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75\" (UID: \"1b3a84af-dd51-40b7-b426-b51d208ca058\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" Oct 30 17:25:34.667975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.666815 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3a84af-dd51-40b7-b426-b51d208ca058-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75\" (UID: \"1b3a84af-dd51-40b7-b426-b51d208ca058\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" Oct 30 17:25:34.667975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.667043 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6cfq\" (UniqueName: \"kubernetes.io/projected/1b3a84af-dd51-40b7-b426-b51d208ca058-kube-api-access-n6cfq\") pod \"payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75\" (UID: \"1b3a84af-dd51-40b7-b426-b51d208ca058\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" Oct 30 17:25:34.692924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.668485 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3a84af-dd51-40b7-b426-b51d208ca058-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75\" (UID: \"1b3a84af-dd51-40b7-b426-b51d208ca058\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" Oct 30 17:25:34.729784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.696648 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7"] Oct 30 17:25:34.817124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.803474 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:34.899316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.899278 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7"] Oct 30 17:25:34.902421 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.899628 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" Oct 30 17:25:34.915423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.915063 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6cfq\" (UniqueName: \"kubernetes.io/projected/1b3a84af-dd51-40b7-b426-b51d208ca058-kube-api-access-n6cfq\") pod \"payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75\" (UID: \"1b3a84af-dd51-40b7-b426-b51d208ca058\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" Oct 30 17:25:34.986124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.986083 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpjmq\" (UniqueName: \"kubernetes.io/projected/f92a7745-1da0-468d-bbb0-fba201967378-kube-api-access-kpjmq\") pod \"payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7\" (UID: \"f92a7745-1da0-468d-bbb0-fba201967378\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" Oct 30 17:25:34.986336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:34.986313 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92a7745-1da0-468d-bbb0-fba201967378-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7\" (UID: \"f92a7745-1da0-468d-bbb0-fba201967378\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" Oct 30 17:25:35.088205 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:35.088158 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92a7745-1da0-468d-bbb0-fba201967378-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7\" (UID: \"f92a7745-1da0-468d-bbb0-fba201967378\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" Oct 30 17:25:35.088416 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:35.088328 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpjmq\" (UniqueName: \"kubernetes.io/projected/f92a7745-1da0-468d-bbb0-fba201967378-kube-api-access-kpjmq\") pod \"payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7\" (UID: \"f92a7745-1da0-468d-bbb0-fba201967378\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" Oct 30 17:25:35.088863 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:35.088841 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92a7745-1da0-468d-bbb0-fba201967378-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7\" (UID: \"f92a7745-1da0-468d-bbb0-fba201967378\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" Oct 30 17:25:35.113046 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:35.113010 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpjmq\" (UniqueName: \"kubernetes.io/projected/f92a7745-1da0-468d-bbb0-fba201967378-kube-api-access-kpjmq\") pod \"payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7\" (UID: \"f92a7745-1da0-468d-bbb0-fba201967378\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" Oct 30 17:25:35.120792 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:35.120763 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" Oct 30 17:25:35.232203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:35.232166 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" Oct 30 17:25:35.436807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:35.436781 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75"] Oct 30 17:25:35.561019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:35.560990 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7"] Oct 30 17:25:36.451075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:36.451031 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" event={"ID":"1b3a84af-dd51-40b7-b426-b51d208ca058","Type":"ContainerStarted","Data":"12b5af56d271a5756c880e8d3d994946d7935d9814a0237f0689a39932105b93"} Oct 30 17:25:36.451572 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:36.451234 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" Oct 30 17:25:36.451572 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:36.451271 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" event={"ID":"1b3a84af-dd51-40b7-b426-b51d208ca058","Type":"ContainerStarted","Data":"437e6c43b02d97c458281989d7c4cd2ac339106376fe438fceda3fb7496bb4cc"} Oct 30 17:25:36.453602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:36.453570 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" event={"ID":"f92a7745-1da0-468d-bbb0-fba201967378","Type":"ContainerStarted","Data":"a8fbf4f4ab82eb5989f21c6e5feae7e9004924e09567bd07d3db734415b2b95f"} Oct 30 17:25:36.453731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:36.453621 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" event={"ID":"f92a7745-1da0-468d-bbb0-fba201967378","Type":"ContainerStarted","Data":"469ba3e2f9621f97eded072ff5372692c7e47b7b39b10f758648ccca0f5f69c4"} Oct 30 17:25:36.453731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:36.453673 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" Oct 30 17:25:36.495771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:36.495718 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" podStartSLOduration=2.495703331 podStartE2EDuration="2.495703331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:36.492863784 +0000 UTC m=+2314.138373963" watchObservedRunningTime="2026-10-30 17:25:36.495703331 +0000 UTC m=+2314.141213511" Oct 30 17:25:36.543119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:36.542156 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" podStartSLOduration=2.5421045319999998 podStartE2EDuration="2.542104532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:36.53709635 +0000 UTC m=+2314.182606533" watchObservedRunningTime="2026-10-30 17:25:36.542104532 +0000 UTC m=+2314.187614713" Oct 30 17:25:36.612441 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:36.612394 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:37.913574 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:25:37.913528 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:25:38.260981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:38.259806 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:39.925933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:39.907262 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:41.635926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:41.632240 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:43.044866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:43.044829 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:43.732544 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:43.732508 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:43.893924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:43.893561 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:44.048604 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:44.047439 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:44.384869 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:44.384769 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:45.650853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:45.650813 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:47.255828 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:47.255786 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:47.465406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:47.465372 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" Oct 30 17:25:47.465617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:47.465452 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" Oct 30 17:25:47.656338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:47.656296 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:49.263400 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:49.263361 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:50.769157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:50.769123 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:50.928625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:50.928588 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:51.034336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:51.034235 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm"] Oct 30 17:25:51.034792 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:51.034757 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" podUID="ce173a53-c21d-4b94-a7ec-d688705efe9e" containerName="istio-proxy" containerID="cri-o://894f78214b39f94ffc7bdd0841e4e95f6c0dab0d20280aa6a65e947f43129ecb" gracePeriod=30 Oct 30 17:25:51.100233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:51.100198 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:52.810552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:52.804374 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:55.439923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:55.434983 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:25:55.455025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:55.454007 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:25:55.455025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:55.454720 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:25:55.769748 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:55.769693 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:55.952558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:55.952509 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:56.100719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.100684 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:56.342520 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.342490 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:25:56.370318 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.370285 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-envoy\") pod \"ce173a53-c21d-4b94-a7ec-d688705efe9e\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " Oct 30 17:25:56.370517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.370325 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce173a53-c21d-4b94-a7ec-d688705efe9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce173a53-c21d-4b94-a7ec-d688705efe9e-istiod-ca-cert\") pod \"ce173a53-c21d-4b94-a7ec-d688705efe9e\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " Oct 30 17:25:56.370517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.370388 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-socket\") pod \"ce173a53-c21d-4b94-a7ec-d688705efe9e\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " Oct 30 17:25:56.370517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.370460 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-data\") pod \"ce173a53-c21d-4b94-a7ec-d688705efe9e\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " Oct 30 17:25:56.370774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.370528 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-kube-api-access-bpzsl\" (UniqueName: \"kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-kube-api-access-bpzsl\") pod \"ce173a53-c21d-4b94-a7ec-d688705efe9e\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " Oct 30 17:25:56.370774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.370664 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-credential-socket\") pod \"ce173a53-c21d-4b94-a7ec-d688705efe9e\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " Oct 30 17:25:56.370774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.370702 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-token\" (UniqueName: \"kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-token\") pod \"ce173a53-c21d-4b94-a7ec-d688705efe9e\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " Oct 30 17:25:56.370774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.370758 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-podinfo\") pod \"ce173a53-c21d-4b94-a7ec-d688705efe9e\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " Oct 30 17:25:56.371035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.370842 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-certs\") pod \"ce173a53-c21d-4b94-a7ec-d688705efe9e\" (UID: \"ce173a53-c21d-4b94-a7ec-d688705efe9e\") " Oct 30 17:25:56.371035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.370695 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-socket" pod "ce173a53-c21d-4b94-a7ec-d688705efe9e" (UID: "ce173a53-c21d-4b94-a7ec-d688705efe9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:56.371035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.370786 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce173a53-c21d-4b94-a7ec-d688705efe9e-istiod-ca-cert" pod "ce173a53-c21d-4b94-a7ec-d688705efe9e" (UID: "ce173a53-c21d-4b94-a7ec-d688705efe9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:56.371266 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.371232 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-credential-socket" pod "ce173a53-c21d-4b94-a7ec-d688705efe9e" (UID: "ce173a53-c21d-4b94-a7ec-d688705efe9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:56.371417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.371330 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-data" pod "ce173a53-c21d-4b94-a7ec-d688705efe9e" (UID: "ce173a53-c21d-4b94-a7ec-d688705efe9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:56.371483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.371425 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-certs" pod "ce173a53-c21d-4b94-a7ec-d688705efe9e" (UID: "ce173a53-c21d-4b94-a7ec-d688705efe9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:56.371751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.371726 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.371864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.371755 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.371864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.371771 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce173a53-c21d-4b94-a7ec-d688705efe9e-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.371864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.371784 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.371864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.371797 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.373028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.373000 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-envoy" pod "ce173a53-c21d-4b94-a7ec-d688705efe9e" (UID: "ce173a53-c21d-4b94-a7ec-d688705efe9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:56.373383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.373355 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-token" pod "ce173a53-c21d-4b94-a7ec-d688705efe9e" (UID: "ce173a53-c21d-4b94-a7ec-d688705efe9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:56.373493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.373421 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-podinfo" pod "ce173a53-c21d-4b94-a7ec-d688705efe9e" (UID: "ce173a53-c21d-4b94-a7ec-d688705efe9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:25:56.373561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.373529 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-kube-api-access-bpzsl" pod "ce173a53-c21d-4b94-a7ec-d688705efe9e" (UID: "ce173a53-c21d-4b94-a7ec-d688705efe9e"). InnerVolumeSpecName "kube-api-access-bpzsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:56.473025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.472993 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.473025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.473021 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpzsl\" (UniqueName: \"kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-kube-api-access-bpzsl\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.473025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.473031 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.473493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.473041 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce173a53-c21d-4b94-a7ec-d688705efe9e-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:25:56.883799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.883767 4828 generic.go:356] "Generic (PLEG): container finished" podID="ce173a53-c21d-4b94-a7ec-d688705efe9e" containerID="894f78214b39f94ffc7bdd0841e4e95f6c0dab0d20280aa6a65e947f43129ecb" exitCode=0 Oct 30 17:25:56.884034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.883818 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" Oct 30 17:25:56.884034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.883835 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" event={"ID":"ce173a53-c21d-4b94-a7ec-d688705efe9e","Type":"ContainerDied","Data":"894f78214b39f94ffc7bdd0841e4e95f6c0dab0d20280aa6a65e947f43129ecb"} Oct 30 17:25:56.884034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.883877 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm" event={"ID":"ce173a53-c21d-4b94-a7ec-d688705efe9e","Type":"ContainerDied","Data":"b171c61dd9adaf3c3ad622b98b186677f64c0a3950e51069a854b7e478ac0361"} Oct 30 17:25:56.884034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.883914 4828 scope.go:122] "RemoveContainer" containerID="894f78214b39f94ffc7bdd0841e4e95f6c0dab0d20280aa6a65e947f43129ecb" Oct 30 17:25:56.896123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.896097 4828 scope.go:122] "RemoveContainer" containerID="894f78214b39f94ffc7bdd0841e4e95f6c0dab0d20280aa6a65e947f43129ecb" Oct 30 17:25:56.896465 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:25:56.896441 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894f78214b39f94ffc7bdd0841e4e95f6c0dab0d20280aa6a65e947f43129ecb\": container with ID starting with 894f78214b39f94ffc7bdd0841e4e95f6c0dab0d20280aa6a65e947f43129ecb not found: ID does not exist" containerID="894f78214b39f94ffc7bdd0841e4e95f6c0dab0d20280aa6a65e947f43129ecb" Oct 30 17:25:56.896562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.896474 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894f78214b39f94ffc7bdd0841e4e95f6c0dab0d20280aa6a65e947f43129ecb"} err="failed to get container status \"894f78214b39f94ffc7bdd0841e4e95f6c0dab0d20280aa6a65e947f43129ecb\": rpc error: code = NotFound desc = could not find container \"894f78214b39f94ffc7bdd0841e4e95f6c0dab0d20280aa6a65e947f43129ecb\": container with ID starting with 894f78214b39f94ffc7bdd0841e4e95f6c0dab0d20280aa6a65e947f43129ecb not found: ID does not exist" Oct 30 17:25:56.943848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.943779 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm"] Oct 30 17:25:56.954009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:56.953963 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-42d381-openshift-default-b7ccf8747-p28jm"] Oct 30 17:25:57.686812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:57.686773 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:57.842339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:57.842300 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8650-0edcad"] Oct 30 17:25:57.843098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:57.843081 4828 cpu_manager.go:414] "containerMap: removing container" podUID="ce173a53-c21d-4b94-a7ec-d688705efe9e" containerName="istio-proxy" Oct 30 17:25:57.843164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:57.843098 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce173a53-c21d-4b94-a7ec-d688705efe9e" containerName="istio-proxy" Oct 30 17:25:57.843224 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:57.843215 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce173a53-c21d-4b94-a7ec-d688705efe9e" containerName="istio-proxy" Oct 30 17:25:57.848452 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:57.848432 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8650-0edcad" Oct 30 17:25:57.863125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:57.863097 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8650-0edcad"] Oct 30 17:25:57.870448 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:57.870426 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wddsm\"" Oct 30 17:25:57.918646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:57.918605 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75xg5\" (UniqueName: \"kubernetes.io/projected/881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c-kube-api-access-75xg5\") pod \"test-curl-8650-0edcad\" (UID: \"881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c\") " pod="opendatahub/test-curl-8650-0edcad" Oct 30 17:25:58.019701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:58.019663 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xg5\" (UniqueName: \"kubernetes.io/projected/881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c-kube-api-access-75xg5\") pod \"test-curl-8650-0edcad\" (UID: \"881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c\") " pod="opendatahub/test-curl-8650-0edcad" Oct 30 17:25:58.044094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:58.044055 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75xg5\" (UniqueName: \"kubernetes.io/projected/881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c-kube-api-access-75xg5\") pod \"test-curl-8650-0edcad\" (UID: \"881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c\") " pod="opendatahub/test-curl-8650-0edcad" Oct 30 17:25:58.162140 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:58.162094 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8650-0edcad" Oct 30 17:25:58.473870 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:58.473836 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8650-0edcad"] Oct 30 17:25:58.574177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:58.574096 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce173a53-c21d-4b94-a7ec-d688705efe9e" path="/var/lib/kubelet/pods/ce173a53-c21d-4b94-a7ec-d688705efe9e/volumes" Oct 30 17:25:58.617719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:58.617692 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:58.908029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:58.907986 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8650-0edcad" event={"ID":"881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c","Type":"ContainerStarted","Data":"88b422dd8068015307d00ddd299e5bd73c791af11c1ec31d33ad4b28c2e44abd"} Oct 30 17:25:58.908029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:58.908044 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8650-0edcad" event={"ID":"881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c","Type":"ContainerStarted","Data":"bdbe0118a8b00905a9913bc49e93c52501ca8d837bbf6c9f4c438045c15e1769"} Oct 30 17:25:58.947686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:58.947631 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8650-0edcad" podStartSLOduration=1.947618581 podStartE2EDuration="1.947618581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:58.947115849 +0000 UTC m=+2336.592626030" watchObservedRunningTime="2026-10-30 17:25:58.947618581 +0000 UTC m=+2336.593128760" Oct 30 17:25:59.377923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:59.375616 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:59.528844 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:59.528810 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:25:59.872660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:59.872626 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8650-0edcad"] Oct 30 17:25:59.878188 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:25:59.878157 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8650-0edcad"] Oct 30 17:26:00.036385 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.036349 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8650-476664"] Oct 30 17:26:00.037167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.037148 4828 cpu_manager.go:414] "containerMap: removing container" podUID="881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c" containerName="test-curl-8650-0edcad" Oct 30 17:26:00.037249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.037169 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c" containerName="test-curl-8650-0edcad" Oct 30 17:26:00.037308 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.037296 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c" containerName="test-curl-8650-0edcad" Oct 30 17:26:00.041051 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.041034 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8650-476664" Oct 30 17:26:00.060312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.060276 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8650-476664"] Oct 30 17:26:00.162417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.162335 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnjzt\" (UniqueName: \"kubernetes.io/projected/2e78f76b-c16b-491b-82b9-f7adc4aed4e8-kube-api-access-fnjzt\") pod \"test-curl-8650-476664\" (UID: \"2e78f76b-c16b-491b-82b9-f7adc4aed4e8\") " pod="opendatahub/test-curl-8650-476664" Oct 30 17:26:00.263757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.263716 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnjzt\" (UniqueName: \"kubernetes.io/projected/2e78f76b-c16b-491b-82b9-f7adc4aed4e8-kube-api-access-fnjzt\") pod \"test-curl-8650-476664\" (UID: \"2e78f76b-c16b-491b-82b9-f7adc4aed4e8\") " pod="opendatahub/test-curl-8650-476664" Oct 30 17:26:00.288228 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.288185 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnjzt\" (UniqueName: \"kubernetes.io/projected/2e78f76b-c16b-491b-82b9-f7adc4aed4e8-kube-api-access-fnjzt\") pod \"test-curl-8650-476664\" (UID: \"2e78f76b-c16b-491b-82b9-f7adc4aed4e8\") " pod="opendatahub/test-curl-8650-476664" Oct 30 17:26:00.355355 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.355321 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8650-476664" Oct 30 17:26:00.673145 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:26:00.673114 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e78f76b_c16b_491b_82b9_f7adc4aed4e8.slice/crio-16af4692296ef6cc3d099961bd350860b30c6681cf4280b390618e05f0ac53a2 WatchSource:0}: Error finding container 16af4692296ef6cc3d099961bd350860b30c6681cf4280b390618e05f0ac53a2: Status 404 returned error can't find the container with id 16af4692296ef6cc3d099961bd350860b30c6681cf4280b390618e05f0ac53a2 Oct 30 17:26:00.684867 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.684832 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8650-476664"] Oct 30 17:26:00.930836 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.930735 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8650-476664" event={"ID":"2e78f76b-c16b-491b-82b9-f7adc4aed4e8","Type":"ContainerStarted","Data":"7353ed20e2401a96443a0d334eb60bce5c8d33f1045cffafb390c3573b042269"} Oct 30 17:26:00.930836 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.930786 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8650-476664" event={"ID":"2e78f76b-c16b-491b-82b9-f7adc4aed4e8","Type":"ContainerStarted","Data":"16af4692296ef6cc3d099961bd350860b30c6681cf4280b390618e05f0ac53a2"} Oct 30 17:26:00.931121 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.930985 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8650-0edcad" podUID="881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c" containerName="test-curl-8650-0edcad" containerID="cri-o://88b422dd8068015307d00ddd299e5bd73c791af11c1ec31d33ad4b28c2e44abd" gracePeriod=2 Oct 30 17:26:00.975754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:00.975688 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8650-476664" podStartSLOduration=0.975667903 podStartE2EDuration="975.667903ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:00.970004619 +0000 UTC m=+2338.615514796" watchObservedRunningTime="2026-10-30 17:26:00.975667903 +0000 UTC m=+2338.621178083" Oct 30 17:26:01.201977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:01.195747 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:01.849725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:01.849685 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb"] Oct 30 17:26:01.922703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:01.922649 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8650-476664"] Oct 30 17:26:01.929657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:01.928446 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8650-476664"] Oct 30 17:26:02.040852 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:02.040808 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:02.199333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:02.199226 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:02.954348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:02.954296 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8650-476664" podUID="2e78f76b-c16b-491b-82b9-f7adc4aed4e8" containerName="test-curl-8650-476664" containerID="cri-o://7353ed20e2401a96443a0d334eb60bce5c8d33f1045cffafb390c3573b042269" gracePeriod=2 Oct 30 17:26:03.064146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:03.064113 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:03.253961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:03.253916 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8650-0edcad" Oct 30 17:26:03.380534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:03.380485 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c-kube-api-access-75xg5\" (UniqueName: \"kubernetes.io/projected/881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c-kube-api-access-75xg5\") pod \"881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c\" (UID: \"881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c\") " Oct 30 17:26:03.382708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:03.382671 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c-kube-api-access-75xg5" pod "881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c" (UID: "881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c"). InnerVolumeSpecName "kube-api-access-75xg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:03.483512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:03.483431 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75xg5\" (UniqueName: \"kubernetes.io/projected/881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c-kube-api-access-75xg5\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:03.963114 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:03.963081 4828 generic.go:356] "Generic (PLEG): container finished" podID="881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c" containerID="88b422dd8068015307d00ddd299e5bd73c791af11c1ec31d33ad4b28c2e44abd" exitCode=137 Oct 30 17:26:03.963571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:03.963127 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8650-0edcad" Oct 30 17:26:03.963571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:03.963183 4828 scope.go:122] "RemoveContainer" containerID="88b422dd8068015307d00ddd299e5bd73c791af11c1ec31d33ad4b28c2e44abd" Oct 30 17:26:03.975179 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:03.975145 4828 scope.go:122] "RemoveContainer" containerID="88b422dd8068015307d00ddd299e5bd73c791af11c1ec31d33ad4b28c2e44abd" Oct 30 17:26:03.975498 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:26:03.975472 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b422dd8068015307d00ddd299e5bd73c791af11c1ec31d33ad4b28c2e44abd\": container with ID starting with 88b422dd8068015307d00ddd299e5bd73c791af11c1ec31d33ad4b28c2e44abd not found: ID does not exist" containerID="88b422dd8068015307d00ddd299e5bd73c791af11c1ec31d33ad4b28c2e44abd" Oct 30 17:26:03.975559 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:03.975510 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b422dd8068015307d00ddd299e5bd73c791af11c1ec31d33ad4b28c2e44abd"} err="failed to get container status \"88b422dd8068015307d00ddd299e5bd73c791af11c1ec31d33ad4b28c2e44abd\": rpc error: code = NotFound desc = could not find container \"88b422dd8068015307d00ddd299e5bd73c791af11c1ec31d33ad4b28c2e44abd\": container with ID starting with 88b422dd8068015307d00ddd299e5bd73c791af11c1ec31d33ad4b28c2e44abd not found: ID does not exist" Oct 30 17:26:03.986820 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:26:03.986776 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8650-0edcad\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c" pod="opendatahub/test-curl-8650-0edcad" Oct 30 17:26:04.512227 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:04.512191 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c" path="/var/lib/kubelet/pods/881157b9-c47a-4d7f-9ae6-a8cc02e3ba4c/volumes" Oct 30 17:26:04.670993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:04.667947 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:05.201560 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:05.201531 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8650-476664" Oct 30 17:26:05.316763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:05.316728 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e78f76b-c16b-491b-82b9-f7adc4aed4e8-kube-api-access-fnjzt\" (UniqueName: \"kubernetes.io/projected/2e78f76b-c16b-491b-82b9-f7adc4aed4e8-kube-api-access-fnjzt\") pod \"2e78f76b-c16b-491b-82b9-f7adc4aed4e8\" (UID: \"2e78f76b-c16b-491b-82b9-f7adc4aed4e8\") " Oct 30 17:26:05.319025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:05.318986 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e78f76b-c16b-491b-82b9-f7adc4aed4e8-kube-api-access-fnjzt" pod "2e78f76b-c16b-491b-82b9-f7adc4aed4e8" (UID: "2e78f76b-c16b-491b-82b9-f7adc4aed4e8"). InnerVolumeSpecName "kube-api-access-fnjzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:05.419595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:05.419552 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnjzt\" (UniqueName: \"kubernetes.io/projected/2e78f76b-c16b-491b-82b9-f7adc4aed4e8-kube-api-access-fnjzt\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:05.983893 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:05.983846 4828 generic.go:356] "Generic (PLEG): container finished" podID="2e78f76b-c16b-491b-82b9-f7adc4aed4e8" containerID="7353ed20e2401a96443a0d334eb60bce5c8d33f1045cffafb390c3573b042269" exitCode=137 Oct 30 17:26:05.983893 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:05.983926 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8650-476664" Oct 30 17:26:05.984224 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:05.983982 4828 scope.go:122] "RemoveContainer" containerID="7353ed20e2401a96443a0d334eb60bce5c8d33f1045cffafb390c3573b042269" Oct 30 17:26:06.000983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:06.000952 4828 scope.go:122] "RemoveContainer" containerID="7353ed20e2401a96443a0d334eb60bce5c8d33f1045cffafb390c3573b042269" Oct 30 17:26:06.001418 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:26:06.001386 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7353ed20e2401a96443a0d334eb60bce5c8d33f1045cffafb390c3573b042269\": container with ID starting with 7353ed20e2401a96443a0d334eb60bce5c8d33f1045cffafb390c3573b042269 not found: ID does not exist" containerID="7353ed20e2401a96443a0d334eb60bce5c8d33f1045cffafb390c3573b042269" Oct 30 17:26:06.001552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:06.001427 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7353ed20e2401a96443a0d334eb60bce5c8d33f1045cffafb390c3573b042269"} err="failed to get container status \"7353ed20e2401a96443a0d334eb60bce5c8d33f1045cffafb390c3573b042269\": rpc error: code = NotFound desc = could not find container \"7353ed20e2401a96443a0d334eb60bce5c8d33f1045cffafb390c3573b042269\": container with ID starting with 7353ed20e2401a96443a0d334eb60bce5c8d33f1045cffafb390c3573b042269 not found: ID does not exist" Oct 30 17:26:06.010134 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:26:06.010086 4828 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8650-476664\" is forbidden: User \"system:node:ip-10-0-135-153.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-153.ec2.internal' and this object" podUID="2e78f76b-c16b-491b-82b9-f7adc4aed4e8" pod="opendatahub/test-curl-8650-476664" Oct 30 17:26:06.450602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:06.450562 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:06.521847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:06.521707 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e78f76b-c16b-491b-82b9-f7adc4aed4e8" path="/var/lib/kubelet/pods/2e78f76b-c16b-491b-82b9-f7adc4aed4e8/volumes" Oct 30 17:26:06.850191 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:06.850152 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" podUID="0b0c1cf4-fc08-47bb-b0a4-493f89042c1e" containerName="payload-processing" containerID="cri-o://49f9a1eb6acd01d121d64439d094f0a123857e1d20295c3c2af5546351febf68" gracePeriod=25 Oct 30 17:26:07.000709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:07.000459 4828 generic.go:356] "Generic (PLEG): container finished" podID="0b0c1cf4-fc08-47bb-b0a4-493f89042c1e" containerID="49f9a1eb6acd01d121d64439d094f0a123857e1d20295c3c2af5546351febf68" exitCode=0 Oct 30 17:26:07.000709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:07.000634 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" event={"ID":"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e","Type":"ContainerDied","Data":"49f9a1eb6acd01d121d64439d094f0a123857e1d20295c3c2af5546351febf68"} Oct 30 17:26:07.150588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:07.150559 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" Oct 30 17:26:07.246686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:07.246647 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-kube-api-access-6dwjp\" (UniqueName: \"kubernetes.io/projected/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-kube-api-access-6dwjp\") pod \"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e\" (UID: \"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e\") " Oct 30 17:26:07.246893 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:07.246840 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-plugins-config-volume\") pod \"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e\" (UID: \"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e\") " Oct 30 17:26:07.247342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:07.247310 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-plugins-config-volume" pod "0b0c1cf4-fc08-47bb-b0a4-493f89042c1e" (UID: "0b0c1cf4-fc08-47bb-b0a4-493f89042c1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:07.247564 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:07.247543 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:07.249607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:07.249570 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-kube-api-access-6dwjp" pod "0b0c1cf4-fc08-47bb-b0a4-493f89042c1e" (UID: "0b0c1cf4-fc08-47bb-b0a4-493f89042c1e"). InnerVolumeSpecName "kube-api-access-6dwjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:07.349156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:07.349117 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwjp\" (UniqueName: \"kubernetes.io/projected/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e-kube-api-access-6dwjp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:07.913223 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:26:07.913186 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:26:08.012444 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:08.012410 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" Oct 30 17:26:08.012656 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:08.012446 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb" event={"ID":"0b0c1cf4-fc08-47bb-b0a4-493f89042c1e","Type":"ContainerDied","Data":"cb4f034d3bc48e15ec85c2fd717dfed9971fa0615ddd94b5be845db6e7983a29"} Oct 30 17:26:08.012656 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:08.012519 4828 scope.go:122] "RemoveContainer" containerID="49f9a1eb6acd01d121d64439d094f0a123857e1d20295c3c2af5546351febf68" Oct 30 17:26:08.083927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:08.083432 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:08.130961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:08.130920 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb"] Oct 30 17:26:08.137888 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:08.137848 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c45877-6fb8f8b446-j6rtb"] Oct 30 17:26:08.493294 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:08.493258 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0c1cf4-fc08-47bb-b0a4-493f89042c1e" path="/var/lib/kubelet/pods/0b0c1cf4-fc08-47bb-b0a4-493f89042c1e/volumes" Oct 30 17:26:09.487442 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.487399 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84"] Oct 30 17:26:09.488384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.488365 4828 cpu_manager.go:414] "containerMap: removing container" podUID="0b0c1cf4-fc08-47bb-b0a4-493f89042c1e" containerName="payload-processing" Oct 30 17:26:09.488447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.488386 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0c1cf4-fc08-47bb-b0a4-493f89042c1e" containerName="payload-processing" Oct 30 17:26:09.488447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.488428 4828 cpu_manager.go:414] "containerMap: removing container" podUID="2e78f76b-c16b-491b-82b9-f7adc4aed4e8" containerName="test-curl-8650-476664" Oct 30 17:26:09.488447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.488434 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e78f76b-c16b-491b-82b9-f7adc4aed4e8" containerName="test-curl-8650-476664" Oct 30 17:26:09.488582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.488572 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e78f76b-c16b-491b-82b9-f7adc4aed4e8" containerName="test-curl-8650-476664" Oct 30 17:26:09.488617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.488592 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0c1cf4-fc08-47bb-b0a4-493f89042c1e" containerName="payload-processing" Oct 30 17:26:09.493907 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.493873 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:09.512252 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.510081 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-469ad7-serving-cert\"" Oct 30 17:26:09.519372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.519340 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84"] Oct 30 17:26:09.593705 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.593665 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b9082e-ac02-4490-a9aa-d805d7f9d737-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:09.593943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.593734 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g6kt\" (UniqueName: \"kubernetes.io/projected/93b9082e-ac02-4490-a9aa-d805d7f9d737-kube-api-access-2g6kt\") pod \"maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:09.593943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.593871 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:09.594115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.594095 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:09.609216 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.609161 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:09.695351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.695308 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:09.695555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.695377 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b9082e-ac02-4490-a9aa-d805d7f9d737-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:09.695555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.695431 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g6kt\" (UniqueName: \"kubernetes.io/projected/93b9082e-ac02-4490-a9aa-d805d7f9d737-kube-api-access-2g6kt\") pod \"maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:09.695555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.695475 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:09.696345 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.696312 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b9082e-ac02-4490-a9aa-d805d7f9d737-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:09.698189 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.698168 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:09.698453 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.698433 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:09.769729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.760623 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g6kt\" (UniqueName: \"kubernetes.io/projected/93b9082e-ac02-4490-a9aa-d805d7f9d737-kube-api-access-2g6kt\") pod \"maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:09.792025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.783865 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:09.826546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:09.826478 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:10.143836 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:26:10.143791 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b9082e_ac02_4490_a9aa_d805d7f9d737.slice/crio-9bcaa1b70d6426f90f295ada0359ba3cf9153dc9b82c3dd58ef234db764a8069 WatchSource:0}: Error finding container 9bcaa1b70d6426f90f295ada0359ba3cf9153dc9b82c3dd58ef234db764a8069: Status 404 returned error can't find the container with id 9bcaa1b70d6426f90f295ada0359ba3cf9153dc9b82c3dd58ef234db764a8069 Oct 30 17:26:10.144531 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:10.144501 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84"] Oct 30 17:26:10.145542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:10.145521 4828 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:26:10.186322 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:10.186284 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps"] Oct 30 17:26:10.258781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:10.258748 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:11.040066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:11.040027 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" event={"ID":"93b9082e-ac02-4490-a9aa-d805d7f9d737","Type":"ContainerStarted","Data":"b90658b778c40e407d8f54f499960045d86c839bd376f8e118a97a2d1f88adb1"} Oct 30 17:26:11.040507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:11.040101 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:11.040507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:11.040121 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" event={"ID":"93b9082e-ac02-4490-a9aa-d805d7f9d737","Type":"ContainerStarted","Data":"9bcaa1b70d6426f90f295ada0359ba3cf9153dc9b82c3dd58ef234db764a8069"} Oct 30 17:26:11.087114 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:11.087057 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" podStartSLOduration=1.676577622 podStartE2EDuration="2.08704132s" totalImagesPullingTime="410.463698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:11.08296719 +0000 UTC m=+2348.728477370" watchObservedRunningTime="2026-10-30 17:26:11.08704132 +0000 UTC m=+2348.732551502" Oct 30 17:26:11.547988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:11.545835 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:13.172920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:13.172581 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:14.758925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:14.758048 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:16.521339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:16.521298 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:17.054488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:17.054458 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:26:18.283230 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:18.283180 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:19.981545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:19.974588 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:21.413394 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.413353 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp"] Oct 30 17:26:21.421095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.421069 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.436844 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.436822 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-5d8947-openshift-default-dockercfg-kfm5z\"" Oct 30 17:26:21.439826 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.439796 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-socket\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.439996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.439862 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-token\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.440065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.440050 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac5920b-1b21-4328-ae7f-4586183f34c7-istiod-ca-cert\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.440113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.440102 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-certs\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.440205 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.440193 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qknmv\" (UniqueName: \"kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-kube-api-access-qknmv\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.440274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.440252 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-credential-socket\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.440314 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.440294 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-envoy\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.440386 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.440372 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-data\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.440423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.440399 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-podinfo\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.456730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.456693 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp"] Oct 30 17:26:21.541882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.541845 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac5920b-1b21-4328-ae7f-4586183f34c7-istiod-ca-cert\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.542089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.541954 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-certs\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.542089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.542013 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qknmv\" (UniqueName: \"kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-kube-api-access-qknmv\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.542190 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.542171 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-credential-socket\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.542243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.542220 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-envoy\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.542307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.542284 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-data\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.542307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.542294 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-certs\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.542420 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.542312 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-podinfo\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.542420 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.542376 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-socket\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.542538 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.542438 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-token\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.542538 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.542507 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-credential-socket\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.542538 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.542525 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-data\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.542697 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.542619 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac5920b-1b21-4328-ae7f-4586183f34c7-istiod-ca-cert\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.542785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.542766 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-socket\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.544589 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.544567 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-envoy\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.544828 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.544809 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-podinfo\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.566499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.566464 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-token\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.566669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.566521 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qknmv\" (UniqueName: \"kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-kube-api-access-qknmv\") pod \"e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.737116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.737029 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:21.748083 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:21.747980 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:22.130167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:22.130133 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp"] Oct 30 17:26:22.130387 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:26:22.130360 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac5920b_1b21_4328_ae7f_4586183f34c7.slice/crio-af60dd386fffd1ba03ccb9424e2e657e4a937f9b60e0cefabb30bc6fbb1f15ea WatchSource:0}: Error finding container af60dd386fffd1ba03ccb9424e2e657e4a937f9b60e0cefabb30bc6fbb1f15ea: Status 404 returned error can't find the container with id af60dd386fffd1ba03ccb9424e2e657e4a937f9b60e0cefabb30bc6fbb1f15ea Oct 30 17:26:22.133247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:22.133212 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:22.133333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:22.133279 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:22.133333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:22.133313 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:22.149952 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:22.149885 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" event={"ID":"fac5920b-1b21-4328-ae7f-4586183f34c7","Type":"ContainerStarted","Data":"af60dd386fffd1ba03ccb9424e2e657e4a937f9b60e0cefabb30bc6fbb1f15ea"} Oct 30 17:26:23.159663 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:23.159624 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" event={"ID":"fac5920b-1b21-4328-ae7f-4586183f34c7","Type":"ContainerStarted","Data":"51f8767de44343fa43926c0ae7b6cf86e074b039f70332a705549af43e92a034"} Oct 30 17:26:23.222113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:23.222055 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" podStartSLOduration=2.222040824 podStartE2EDuration="2.222040824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:23.219663543 +0000 UTC m=+2360.865173720" watchObservedRunningTime="2026-10-30 17:26:23.222040824 +0000 UTC m=+2360.867551045" Oct 30 17:26:23.431439 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:23.431344 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:23.737801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:23.737705 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:23.740198 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:23.740174 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 17:26:23.740334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:23.740223 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" podUID="fac5920b-1b21-4328-ae7f-4586183f34c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 17:26:24.738344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:24.738305 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 17:26:24.738978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:24.738360 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" podUID="fac5920b-1b21-4328-ae7f-4586183f34c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 17:26:25.218198 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.218156 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" podUID="3c0ea1d5-82de-42de-a926-4c667fe4a066" containerName="main" containerID="cri-o://00e02e86526b4bad934348cec54c97133e49d198fd035557c0ef4e4c638f7b53" gracePeriod=45 Oct 30 17:26:25.294938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.293562 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:25.726254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.726221 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:26:25.737699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.737665 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 17:26:25.737876 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.737750 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" podUID="fac5920b-1b21-4328-ae7f-4586183f34c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 17:26:25.750355 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.750317 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-model-cache\") pod \"3c0ea1d5-82de-42de-a926-4c667fe4a066\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " Oct 30 17:26:25.750835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.750410 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c0ea1d5-82de-42de-a926-4c667fe4a066-kube-api-access-vz47n\" (UniqueName: \"kubernetes.io/projected/3c0ea1d5-82de-42de-a926-4c667fe4a066-kube-api-access-vz47n\") pod \"3c0ea1d5-82de-42de-a926-4c667fe4a066\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " Oct 30 17:26:25.750835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.750563 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-tmp-dir\") pod \"3c0ea1d5-82de-42de-a926-4c667fe4a066\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " Oct 30 17:26:25.750835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.750614 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c0ea1d5-82de-42de-a926-4c667fe4a066-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c0ea1d5-82de-42de-a926-4c667fe4a066-tls-certs\") pod \"3c0ea1d5-82de-42de-a926-4c667fe4a066\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " Oct 30 17:26:25.750835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.750659 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-model-cache" pod "3c0ea1d5-82de-42de-a926-4c667fe4a066" (UID: "3c0ea1d5-82de-42de-a926-4c667fe4a066"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:25.750835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.750747 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-dshm\") pod \"3c0ea1d5-82de-42de-a926-4c667fe4a066\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " Oct 30 17:26:25.750835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.750826 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-home\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-home\") pod \"3c0ea1d5-82de-42de-a926-4c667fe4a066\" (UID: \"3c0ea1d5-82de-42de-a926-4c667fe4a066\") " Oct 30 17:26:25.751210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.750843 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-tmp-dir" pod "3c0ea1d5-82de-42de-a926-4c667fe4a066" (UID: "3c0ea1d5-82de-42de-a926-4c667fe4a066"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:25.751210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.751130 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-home" pod "3c0ea1d5-82de-42de-a926-4c667fe4a066" (UID: "3c0ea1d5-82de-42de-a926-4c667fe4a066"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:25.751547 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.751526 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:25.751624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.751551 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:25.751624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.751564 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:25.752953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.752874 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-dshm" pod "3c0ea1d5-82de-42de-a926-4c667fe4a066" (UID: "3c0ea1d5-82de-42de-a926-4c667fe4a066"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:25.752953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.752885 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c0ea1d5-82de-42de-a926-4c667fe4a066-tls-certs" pod "3c0ea1d5-82de-42de-a926-4c667fe4a066" (UID: "3c0ea1d5-82de-42de-a926-4c667fe4a066"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:25.753095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.752964 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0ea1d5-82de-42de-a926-4c667fe4a066-kube-api-access-vz47n" pod "3c0ea1d5-82de-42de-a926-4c667fe4a066" (UID: "3c0ea1d5-82de-42de-a926-4c667fe4a066"). InnerVolumeSpecName "kube-api-access-vz47n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:25.853187 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.853146 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c0ea1d5-82de-42de-a926-4c667fe4a066-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:25.853187 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.853188 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz47n\" (UniqueName: \"kubernetes.io/projected/3c0ea1d5-82de-42de-a926-4c667fe4a066-kube-api-access-vz47n\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:25.853464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:25.853205 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c0ea1d5-82de-42de-a926-4c667fe4a066-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:26.219107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:26.219010 4828 generic.go:356] "Generic (PLEG): container finished" podID="3c0ea1d5-82de-42de-a926-4c667fe4a066" containerID="00e02e86526b4bad934348cec54c97133e49d198fd035557c0ef4e4c638f7b53" exitCode=0 Oct 30 17:26:26.219275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:26.219258 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" event={"ID":"3c0ea1d5-82de-42de-a926-4c667fe4a066","Type":"ContainerDied","Data":"00e02e86526b4bad934348cec54c97133e49d198fd035557c0ef4e4c638f7b53"} Oct 30 17:26:26.219327 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:26.219297 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" event={"ID":"3c0ea1d5-82de-42de-a926-4c667fe4a066","Type":"ContainerDied","Data":"0476c6ab07d4cf3239edd3b7ce08ba40cf46d228b4ab889854b1b011b4ddbe79"} Oct 30 17:26:26.219327 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:26.219319 4828 scope.go:122] "RemoveContainer" containerID="00e02e86526b4bad934348cec54c97133e49d198fd035557c0ef4e4c638f7b53" Oct 30 17:26:26.219478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:26.219460 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps" Oct 30 17:26:26.233173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:26.233149 4828 scope.go:122] "RemoveContainer" containerID="00e02e86526b4bad934348cec54c97133e49d198fd035557c0ef4e4c638f7b53" Oct 30 17:26:26.233486 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:26:26.233430 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e02e86526b4bad934348cec54c97133e49d198fd035557c0ef4e4c638f7b53\": container with ID starting with 00e02e86526b4bad934348cec54c97133e49d198fd035557c0ef4e4c638f7b53 not found: ID does not exist" containerID="00e02e86526b4bad934348cec54c97133e49d198fd035557c0ef4e4c638f7b53" Oct 30 17:26:26.233486 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:26.233465 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e02e86526b4bad934348cec54c97133e49d198fd035557c0ef4e4c638f7b53"} err="failed to get container status \"00e02e86526b4bad934348cec54c97133e49d198fd035557c0ef4e4c638f7b53\": rpc error: code = NotFound desc = could not find container \"00e02e86526b4bad934348cec54c97133e49d198fd035557c0ef4e4c638f7b53\": container with ID starting with 00e02e86526b4bad934348cec54c97133e49d198fd035557c0ef4e4c638f7b53 not found: ID does not exist" Oct 30 17:26:26.310926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:26.302937 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps"] Oct 30 17:26:26.315277 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:26.315243 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5d8947/auth-test-model-5d8947-kserve-5666f7c8df-fbnps"] Oct 30 17:26:26.511000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:26.509609 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c0ea1d5-82de-42de-a926-4c667fe4a066" path="/var/lib/kubelet/pods/3c0ea1d5-82de-42de-a926-4c667fe4a066/volumes" Oct 30 17:26:26.737810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:26.737775 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 17:26:26.738005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:26.737842 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" podUID="fac5920b-1b21-4328-ae7f-4586183f34c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 17:26:27.382925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:27.380857 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:27.737841 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:27.737738 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 17:26:27.738048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:27.737837 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" podUID="fac5920b-1b21-4328-ae7f-4586183f34c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 17:26:28.676367 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:28.676322 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc"] Oct 30 17:26:28.676932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:28.676594 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" podUID="99acb5e4-3966-4cc9-a242-59c620b065f9" containerName="maas-api" containerID="cri-o://71b1a640f0f2818079bde52e4ef9c8dba7c7f27b67a07076799d119551d43709" gracePeriod=30 Oct 30 17:26:28.738406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:28.738372 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Oct 30 17:26:28.738686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:28.738657 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" podUID="fac5920b-1b21-4328-ae7f-4586183f34c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Oct 30 17:26:28.787125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:28.787028 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:29.011980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.009425 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:29.390655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.390622 4828 generic.go:356] "Generic (PLEG): container finished" podID="99acb5e4-3966-4cc9-a242-59c620b065f9" containerID="71b1a640f0f2818079bde52e4ef9c8dba7c7f27b67a07076799d119551d43709" exitCode=0 Oct 30 17:26:29.391142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.391116 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" event={"ID":"99acb5e4-3966-4cc9-a242-59c620b065f9","Type":"ContainerDied","Data":"71b1a640f0f2818079bde52e4ef9c8dba7c7f27b67a07076799d119551d43709"} Oct 30 17:26:29.470099 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.470061 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:26:29.571336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.571298 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:29.573226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.573195 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99acb5e4-3966-4cc9-a242-59c620b065f9-kube-api-access-fr8gp\" (UniqueName: \"kubernetes.io/projected/99acb5e4-3966-4cc9-a242-59c620b065f9-kube-api-access-fr8gp\") pod \"99acb5e4-3966-4cc9-a242-59c620b065f9\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " Oct 30 17:26:29.573376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.573326 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-maas-api-tls\") pod \"99acb5e4-3966-4cc9-a242-59c620b065f9\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " Oct 30 17:26:29.573376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.573366 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99acb5e4-3966-4cc9-a242-59c620b065f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99acb5e4-3966-4cc9-a242-59c620b065f9-openshift-service-ca\") pod \"99acb5e4-3966-4cc9-a242-59c620b065f9\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " Oct 30 17:26:29.573499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.573452 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-metrics-certs\") pod \"99acb5e4-3966-4cc9-a242-59c620b065f9\" (UID: \"99acb5e4-3966-4cc9-a242-59c620b065f9\") " Oct 30 17:26:29.574368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.574342 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99acb5e4-3966-4cc9-a242-59c620b065f9-openshift-service-ca" pod "99acb5e4-3966-4cc9-a242-59c620b065f9" (UID: "99acb5e4-3966-4cc9-a242-59c620b065f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:29.585764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.585720 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-metrics-certs" pod "99acb5e4-3966-4cc9-a242-59c620b065f9" (UID: "99acb5e4-3966-4cc9-a242-59c620b065f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:29.587659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.587628 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99acb5e4-3966-4cc9-a242-59c620b065f9-kube-api-access-fr8gp" pod "99acb5e4-3966-4cc9-a242-59c620b065f9" (UID: "99acb5e4-3966-4cc9-a242-59c620b065f9"). InnerVolumeSpecName "kube-api-access-fr8gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:29.587801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.587720 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-maas-api-tls" pod "99acb5e4-3966-4cc9-a242-59c620b065f9" (UID: "99acb5e4-3966-4cc9-a242-59c620b065f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:29.676483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.676443 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr8gp\" (UniqueName: \"kubernetes.io/projected/99acb5e4-3966-4cc9-a242-59c620b065f9-kube-api-access-fr8gp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:29.676483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.676481 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:29.677058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.676497 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99acb5e4-3966-4cc9-a242-59c620b065f9-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:29.677058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.676513 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99acb5e4-3966-4cc9-a242-59c620b065f9-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:29.720412 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.720371 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:29.743331 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.743296 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:29.743685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.743663 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:29.744418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.744397 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:26:29.889586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.889500 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:29.982672 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.982627 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d"] Oct 30 17:26:29.984197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.984148 4828 cpu_manager.go:414] "containerMap: removing container" podUID="99acb5e4-3966-4cc9-a242-59c620b065f9" containerName="maas-api" Oct 30 17:26:29.984417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.984398 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99acb5e4-3966-4cc9-a242-59c620b065f9" containerName="maas-api" Oct 30 17:26:29.984564 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.984551 4828 cpu_manager.go:414] "containerMap: removing container" podUID="3c0ea1d5-82de-42de-a926-4c667fe4a066" containerName="main" Oct 30 17:26:29.984660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.984639 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c0ea1d5-82de-42de-a926-4c667fe4a066" containerName="main" Oct 30 17:26:29.984881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.984866 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="99acb5e4-3966-4cc9-a242-59c620b065f9" containerName="maas-api" Oct 30 17:26:29.985015 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.984895 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c0ea1d5-82de-42de-a926-4c667fe4a066" containerName="main" Oct 30 17:26:29.989794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:29.989767 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.001329 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.001286 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-envoy\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.001512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.001354 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-socket\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.001512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.001432 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istiod-ca-cert\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.001651 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.001518 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-credential-socket\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.001651 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.001550 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-certs\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.001651 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.001602 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-token\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.001832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.001656 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k2gs\" (UniqueName: \"kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-kube-api-access-8k2gs\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.001832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.001688 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-podinfo\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.001832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.001767 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-data\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.007605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.006592 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d2d985-openshift-default-dockercfg-z845h\"" Oct 30 17:26:30.014774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.014742 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d"] Oct 30 17:26:30.049969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.049935 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:30.127938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.126955 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-credential-socket\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.127938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.127018 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-certs\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.127938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.127077 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-token\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.127938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.127136 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k2gs\" (UniqueName: \"kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-kube-api-access-8k2gs\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.127938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.127175 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-podinfo\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.127938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.127255 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-data\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.127938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.127378 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-envoy\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.127938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.127423 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-socket\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.127938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.127599 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istiod-ca-cert\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.128708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.128512 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istiod-ca-cert\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.134946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.128804 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-credential-socket\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.134946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.129086 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-certs\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.205988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.197681 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-socket\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.205988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.198233 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-data\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.234666 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.234554 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:30.283969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.283040 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-envoy\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.284354 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.284326 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-token\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.284531 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.284332 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-podinfo\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.306761 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.306719 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k2gs\" (UniqueName: \"kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-kube-api-access-8k2gs\") pod \"e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.330455 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.330347 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:30.404933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.404856 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:30.461761 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.461668 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" Oct 30 17:26:30.463134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.463105 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc" event={"ID":"99acb5e4-3966-4cc9-a242-59c620b065f9","Type":"ContainerDied","Data":"5588d20720e20d9d5be46c444c0d7ae631b225423edf9d2e8fc28dc37a05bcac"} Oct 30 17:26:30.463361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.463344 4828 scope.go:122] "RemoveContainer" containerID="71b1a640f0f2818079bde52e4ef9c8dba7c7f27b67a07076799d119551d43709" Oct 30 17:26:30.525001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.524878 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc"] Oct 30 17:26:30.532236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.532204 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c45877-c5f797c9d-lgmfc"] Oct 30 17:26:30.775945 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:26:30.773052 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f06243_2a82_42f6_aeaa_a187b71b4dab.slice/crio-6c4bcab1101ff58ff96af98b4c35ca76f7c4a3eee57389045d67f0bd31e8a051 WatchSource:0}: Error finding container 6c4bcab1101ff58ff96af98b4c35ca76f7c4a3eee57389045d67f0bd31e8a051: Status 404 returned error can't find the container with id 6c4bcab1101ff58ff96af98b4c35ca76f7c4a3eee57389045d67f0bd31e8a051 Oct 30 17:26:30.776506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.776295 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:30.776506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.776383 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:30.776506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.776426 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:30.778775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:30.778749 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d"] Oct 30 17:26:31.473514 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:31.473462 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" event={"ID":"f3f06243-2a82-42f6-aeaa-a187b71b4dab","Type":"ContainerStarted","Data":"1f94ee67b7770ee0a8c0c2c951a5f0942a9e18b4e64dad5ee154f2d34dd144f0"} Oct 30 17:26:31.473717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:31.473528 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" event={"ID":"f3f06243-2a82-42f6-aeaa-a187b71b4dab","Type":"ContainerStarted","Data":"6c4bcab1101ff58ff96af98b4c35ca76f7c4a3eee57389045d67f0bd31e8a051"} Oct 30 17:26:31.538628 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:31.538573 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" podStartSLOduration=2.538558765 podStartE2EDuration="2.538558765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:31.53429789 +0000 UTC m=+2369.179808072" watchObservedRunningTime="2026-10-30 17:26:31.538558765 +0000 UTC m=+2369.184069018" Oct 30 17:26:31.711788 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:31.711746 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:32.331674 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:32.331637 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:32.338042 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:32.338010 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:32.493835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:32.493797 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99acb5e4-3966-4cc9-a242-59c620b065f9" path="/var/lib/kubelet/pods/99acb5e4-3966-4cc9-a242-59c620b065f9/volumes" Oct 30 17:26:32.494280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:32.494259 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:32.494399 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:32.494348 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:26:33.892890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:33.891398 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:34.087741 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:34.087699 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:34.241547 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:34.241459 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:35.786624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:35.786546 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:35.932425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:35.932385 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:35.947952 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:35.947917 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss"] Oct 30 17:26:35.954062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:35.954035 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" Oct 30 17:26:35.976800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:35.976764 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss"] Oct 30 17:26:35.994216 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:35.994177 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d2d985\"" Oct 30 17:26:35.994417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:35.994385 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d2d985-dockercfg-4p7h4\"" Oct 30 17:26:36.084586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.084448 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45090c4-3935-47e9-acc0-549bc57be3f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss\" (UID: \"b45090c4-3935-47e9-acc0-549bc57be3f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" Oct 30 17:26:36.114087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.114047 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clvqj\" (UniqueName: \"kubernetes.io/projected/b45090c4-3935-47e9-acc0-549bc57be3f3-kube-api-access-clvqj\") pod \"payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss\" (UID: \"b45090c4-3935-47e9-acc0-549bc57be3f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" Oct 30 17:26:36.209925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.203745 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw"] Oct 30 17:26:36.217676 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.217185 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45090c4-3935-47e9-acc0-549bc57be3f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss\" (UID: \"b45090c4-3935-47e9-acc0-549bc57be3f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" Oct 30 17:26:36.218037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.218001 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clvqj\" (UniqueName: \"kubernetes.io/projected/b45090c4-3935-47e9-acc0-549bc57be3f3-kube-api-access-clvqj\") pod \"payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss\" (UID: \"b45090c4-3935-47e9-acc0-549bc57be3f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" Oct 30 17:26:36.220124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.220100 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45090c4-3935-47e9-acc0-549bc57be3f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss\" (UID: \"b45090c4-3935-47e9-acc0-549bc57be3f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" Oct 30 17:26:36.279922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.275426 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" Oct 30 17:26:36.288650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.286999 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clvqj\" (UniqueName: \"kubernetes.io/projected/b45090c4-3935-47e9-acc0-549bc57be3f3-kube-api-access-clvqj\") pod \"payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss\" (UID: \"b45090c4-3935-47e9-acc0-549bc57be3f3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" Oct 30 17:26:36.290224 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.290024 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw"] Oct 30 17:26:36.322954 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.320867 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9qxh\" (UniqueName: \"kubernetes.io/projected/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-kube-api-access-t9qxh\") pod \"payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw\" (UID: \"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" Oct 30 17:26:36.322954 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.321019 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw\" (UID: \"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" Oct 30 17:26:36.394941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.394820 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d"] Oct 30 17:26:36.422127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.422082 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw\" (UID: \"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" Oct 30 17:26:36.422324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.422300 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9qxh\" (UniqueName: \"kubernetes.io/projected/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-kube-api-access-t9qxh\") pod \"payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw\" (UID: \"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" Oct 30 17:26:36.422953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.422888 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw\" (UID: \"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" Oct 30 17:26:36.453937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.446871 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9qxh\" (UniqueName: \"kubernetes.io/projected/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-kube-api-access-t9qxh\") pod \"payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw\" (UID: \"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" Oct 30 17:26:36.569477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.569442 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" Oct 30 17:26:36.580604 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.580570 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:36.649729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.648840 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" Oct 30 17:26:36.778583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.778544 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:36.936945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.936744 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss"] Oct 30 17:26:36.937725 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:26:36.937697 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45090c4_3935_47e9_acc0_549bc57be3f3.slice/crio-cd4565a33d9486d7a8a0323a80fb347fe3cb595bd0c47e8e2345b5a835ff1174 WatchSource:0}: Error finding container cd4565a33d9486d7a8a0323a80fb347fe3cb595bd0c47e8e2345b5a835ff1174: Status 404 returned error can't find the container with id cd4565a33d9486d7a8a0323a80fb347fe3cb595bd0c47e8e2345b5a835ff1174 Oct 30 17:26:36.948479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:36.948456 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:37.261929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:37.257992 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw"] Oct 30 17:26:37.539543 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:37.539501 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" event={"ID":"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f","Type":"ContainerStarted","Data":"75ddd278b68e9af92853c6e9b7588a49b4ee8cfc55053500b3cfce664bf00c68"} Oct 30 17:26:37.539737 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:37.539568 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" event={"ID":"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f","Type":"ContainerStarted","Data":"0e011f997aac02ab266191bbff7837deebe515a5bc6570e4c0c32f9f17e0fcf5"} Oct 30 17:26:37.539737 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:37.539596 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" Oct 30 17:26:37.541182 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:37.541157 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" event={"ID":"b45090c4-3935-47e9-acc0-549bc57be3f3","Type":"ContainerStarted","Data":"c751841da1465605bb34d75f17f9cc89e0cf08c01d16f1953656fc0b6246fd8d"} Oct 30 17:26:37.541311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:37.541208 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" event={"ID":"b45090c4-3935-47e9-acc0-549bc57be3f3","Type":"ContainerStarted","Data":"cd4565a33d9486d7a8a0323a80fb347fe3cb595bd0c47e8e2345b5a835ff1174"} Oct 30 17:26:37.541311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:37.541237 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" Oct 30 17:26:37.588178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:37.588113 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" podStartSLOduration=1.588094945 podStartE2EDuration="1.588094945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:37.581202791 +0000 UTC m=+2375.226712972" watchObservedRunningTime="2026-10-30 17:26:37.588094945 +0000 UTC m=+2375.233605125" Oct 30 17:26:37.623236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:37.623182 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" podStartSLOduration=2.623165557 podStartE2EDuration="2.623165557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:37.622811778 +0000 UTC m=+2375.268321957" watchObservedRunningTime="2026-10-30 17:26:37.623165557 +0000 UTC m=+2375.268675736" Oct 30 17:26:37.808649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:37.808547 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:37.968503 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:37.968464 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:37.991840 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:26:37.991793 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:26:38.093057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:38.092971 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd"] Oct 30 17:26:38.093238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:38.093210 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" podUID="875c10a1-3b01-46e0-a4ed-62e65c551a91" containerName="istio-proxy" containerID="cri-o://aaa0079c18457abb10c352ffe2a134f82b91761c5c205504a3f55b42f24503f4" gracePeriod=30 Oct 30 17:26:38.173096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:38.173057 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:39.027252 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.027206 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:39.404095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.400181 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9"] Oct 30 17:26:39.406495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.406468 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.438598 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.431357 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-b87f3c-openshift-default-dockercfg-zqbjj\"" Oct 30 17:26:39.468029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.448456 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9"] Oct 30 17:26:39.499700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.499660 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:39.571125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.571087 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54e68535-8263-4634-9c81-174743d3de5e-istio-podinfo\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.571324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.571182 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-certs\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.571324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.571245 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-credential-socket\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.571324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.571265 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-envoy\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.571458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.571430 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54e68535-8263-4634-9c81-174743d3de5e-istiod-ca-cert\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.571549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.571517 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-socket\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.571687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.571593 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjszj\" (UniqueName: \"kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-kube-api-access-zjszj\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.571743 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.571700 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-data\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.571819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.571806 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-istio-token\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.653911 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.653854 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:39.675302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.675207 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-data\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.675777 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.675748 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-istio-token\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.676000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.675980 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54e68535-8263-4634-9c81-174743d3de5e-istio-podinfo\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.676161 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.676143 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-certs\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.676337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.676321 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-credential-socket\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.676467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.676453 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-envoy\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.676632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.676615 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54e68535-8263-4634-9c81-174743d3de5e-istiod-ca-cert\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.676756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.676741 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-socket\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.676939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.676919 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjszj\" (UniqueName: \"kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-kube-api-access-zjszj\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.692711 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.692673 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-credential-socket\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.705968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.695746 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54e68535-8263-4634-9c81-174743d3de5e-istiod-ca-cert\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.706203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.696078 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-socket\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.706344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.697375 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-certs\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.706455 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.705280 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-data\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.731822 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.731753 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54e68535-8263-4634-9c81-174743d3de5e-istio-podinfo\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.738934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.733204 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-envoy\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.739680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.739609 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-istio-token\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.764329 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.763829 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjszj\" (UniqueName: \"kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-kube-api-access-zjszj\") pod \"e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:39.834490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:39.833772 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:40.043623 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:40.043581 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:40.454929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:40.450701 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9"] Oct 30 17:26:40.467962 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:26:40.463345 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e68535_8263_4634_9c81_174743d3de5e.slice/crio-4999070a281190fa776957b02713cac6506d71bd5d209c0060db89fb1e8f9ed0 WatchSource:0}: Error finding container 4999070a281190fa776957b02713cac6506d71bd5d209c0060db89fb1e8f9ed0: Status 404 returned error can't find the container with id 4999070a281190fa776957b02713cac6506d71bd5d209c0060db89fb1e8f9ed0 Oct 30 17:26:40.573422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:40.573370 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:40.573727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:40.573700 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:40.573920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:40.573882 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:40.642582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:40.642541 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" event={"ID":"54e68535-8263-4634-9c81-174743d3de5e","Type":"ContainerStarted","Data":"4999070a281190fa776957b02713cac6506d71bd5d209c0060db89fb1e8f9ed0"} Oct 30 17:26:41.397035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:41.396204 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" podUID="b6024600-42ca-45d6-97e5-cc0ae8071f3b" containerName="payload-processing" containerID="cri-o://b4d4020998dc13e277a8293003ac41f92291711ba7086bf4a3c82191d2872949" gracePeriod=25 Oct 30 17:26:41.448344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:41.447247 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:26:41.714823 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:41.714777 4828 generic.go:356] "Generic (PLEG): container finished" podID="b6024600-42ca-45d6-97e5-cc0ae8071f3b" containerID="b4d4020998dc13e277a8293003ac41f92291711ba7086bf4a3c82191d2872949" exitCode=0 Oct 30 17:26:41.715821 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:41.715790 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" event={"ID":"b6024600-42ca-45d6-97e5-cc0ae8071f3b","Type":"ContainerDied","Data":"b4d4020998dc13e277a8293003ac41f92291711ba7086bf4a3c82191d2872949"} Oct 30 17:26:41.734753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:41.734710 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" event={"ID":"54e68535-8263-4634-9c81-174743d3de5e","Type":"ContainerStarted","Data":"f08005d96cef9ec2453eaa4d9fc5db754dc1e35cde3d67e4fc33e6adf487f8bb"} Oct 30 17:26:42.052985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:42.052878 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:42.066700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:42.063664 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:26:42.066700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:42.063729 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:26:42.411662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:42.370678 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:42.428780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:42.428736 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:26:42.457213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:42.457175 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:26:42.598050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:42.597996 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded" Oct 30 17:26:42.912312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:42.896131 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:43.061264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.061182 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:43.124799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.124737 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" podStartSLOduration=4.1247202099999996 podStartE2EDuration="4.12472021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:41.826137674 +0000 UTC m=+2379.471647855" watchObservedRunningTime="2026-10-30 17:26:43.12472021 +0000 UTC m=+2380.770230393" Oct 30 17:26:43.218425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.218382 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" Oct 30 17:26:43.222159 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.222130 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6024600-42ca-45d6-97e5-cc0ae8071f3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6024600-42ca-45d6-97e5-cc0ae8071f3b-plugins-config-volume\") pod \"b6024600-42ca-45d6-97e5-cc0ae8071f3b\" (UID: \"b6024600-42ca-45d6-97e5-cc0ae8071f3b\") " Oct 30 17:26:43.222398 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.222380 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6024600-42ca-45d6-97e5-cc0ae8071f3b-kube-api-access-sggf9\" (UniqueName: \"kubernetes.io/projected/b6024600-42ca-45d6-97e5-cc0ae8071f3b-kube-api-access-sggf9\") pod \"b6024600-42ca-45d6-97e5-cc0ae8071f3b\" (UID: \"b6024600-42ca-45d6-97e5-cc0ae8071f3b\") " Oct 30 17:26:43.222591 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.222562 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6024600-42ca-45d6-97e5-cc0ae8071f3b-plugins-config-volume" pod "b6024600-42ca-45d6-97e5-cc0ae8071f3b" (UID: "b6024600-42ca-45d6-97e5-cc0ae8071f3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:43.223166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.223143 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6024600-42ca-45d6-97e5-cc0ae8071f3b-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.228470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.228440 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6024600-42ca-45d6-97e5-cc0ae8071f3b-kube-api-access-sggf9" pod "b6024600-42ca-45d6-97e5-cc0ae8071f3b" (UID: "b6024600-42ca-45d6-97e5-cc0ae8071f3b"). InnerVolumeSpecName "kube-api-access-sggf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:43.325025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.324916 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sggf9\" (UniqueName: \"kubernetes.io/projected/b6024600-42ca-45d6-97e5-cc0ae8071f3b-kube-api-access-sggf9\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.476925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.476115 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:26:43.529953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.528983 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-envoy\") pod \"875c10a1-3b01-46e0-a4ed-62e65c551a91\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " Oct 30 17:26:43.529953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.529072 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-podinfo\") pod \"875c10a1-3b01-46e0-a4ed-62e65c551a91\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " Oct 30 17:26:43.529953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.529173 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-token\" (UniqueName: \"kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-token\") pod \"875c10a1-3b01-46e0-a4ed-62e65c551a91\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " Oct 30 17:26:43.529953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.529205 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-socket\") pod \"875c10a1-3b01-46e0-a4ed-62e65c551a91\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " Oct 30 17:26:43.529953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.529233 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/875c10a1-3b01-46e0-a4ed-62e65c551a91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/875c10a1-3b01-46e0-a4ed-62e65c551a91-istiod-ca-cert\") pod \"875c10a1-3b01-46e0-a4ed-62e65c551a91\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " Oct 30 17:26:43.529953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.529302 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-kube-api-access-k6xrf\" (UniqueName: \"kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-kube-api-access-k6xrf\") pod \"875c10a1-3b01-46e0-a4ed-62e65c551a91\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " Oct 30 17:26:43.529953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.529381 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-certs\") pod \"875c10a1-3b01-46e0-a4ed-62e65c551a91\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " Oct 30 17:26:43.529953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.529433 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-credential-socket\") pod \"875c10a1-3b01-46e0-a4ed-62e65c551a91\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " Oct 30 17:26:43.529953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.529526 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-data\") pod \"875c10a1-3b01-46e0-a4ed-62e65c551a91\" (UID: \"875c10a1-3b01-46e0-a4ed-62e65c551a91\") " Oct 30 17:26:43.530581 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.530556 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-certs" pod "875c10a1-3b01-46e0-a4ed-62e65c551a91" (UID: "875c10a1-3b01-46e0-a4ed-62e65c551a91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:43.530687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.530603 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-data" pod "875c10a1-3b01-46e0-a4ed-62e65c551a91" (UID: "875c10a1-3b01-46e0-a4ed-62e65c551a91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:43.530817 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.530798 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-credential-socket" pod "875c10a1-3b01-46e0-a4ed-62e65c551a91" (UID: "875c10a1-3b01-46e0-a4ed-62e65c551a91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:43.531048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.531024 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-socket" pod "875c10a1-3b01-46e0-a4ed-62e65c551a91" (UID: "875c10a1-3b01-46e0-a4ed-62e65c551a91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:43.531206 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.531181 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875c10a1-3b01-46e0-a4ed-62e65c551a91-istiod-ca-cert" pod "875c10a1-3b01-46e0-a4ed-62e65c551a91" (UID: "875c10a1-3b01-46e0-a4ed-62e65c551a91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:43.532553 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.532522 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-kube-api-access-k6xrf" pod "875c10a1-3b01-46e0-a4ed-62e65c551a91" (UID: "875c10a1-3b01-46e0-a4ed-62e65c551a91"). InnerVolumeSpecName "kube-api-access-k6xrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:43.533797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.533767 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-token" pod "875c10a1-3b01-46e0-a4ed-62e65c551a91" (UID: "875c10a1-3b01-46e0-a4ed-62e65c551a91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:43.533942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.533841 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-podinfo" pod "875c10a1-3b01-46e0-a4ed-62e65c551a91" (UID: "875c10a1-3b01-46e0-a4ed-62e65c551a91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:26:43.535249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.535222 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-envoy" pod "875c10a1-3b01-46e0-a4ed-62e65c551a91" (UID: "875c10a1-3b01-46e0-a4ed-62e65c551a91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:43.631937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.631815 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.631937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.631850 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.631937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.631862 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.631937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.631883 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.631937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.631891 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.631937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.631911 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.631937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.631920 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/875c10a1-3b01-46e0-a4ed-62e65c551a91-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.631937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.631928 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/875c10a1-3b01-46e0-a4ed-62e65c551a91-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.631937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:43.631938 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6xrf\" (UniqueName: \"kubernetes.io/projected/875c10a1-3b01-46e0-a4ed-62e65c551a91-kube-api-access-k6xrf\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:26:44.034657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.034621 4828 generic.go:356] "Generic (PLEG): container finished" podID="875c10a1-3b01-46e0-a4ed-62e65c551a91" containerID="aaa0079c18457abb10c352ffe2a134f82b91761c5c205504a3f55b42f24503f4" exitCode=0 Oct 30 17:26:44.034871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.034676 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" Oct 30 17:26:44.034871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.034718 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" event={"ID":"875c10a1-3b01-46e0-a4ed-62e65c551a91","Type":"ContainerDied","Data":"aaa0079c18457abb10c352ffe2a134f82b91761c5c205504a3f55b42f24503f4"} Oct 30 17:26:44.034871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.034773 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd" event={"ID":"875c10a1-3b01-46e0-a4ed-62e65c551a91","Type":"ContainerDied","Data":"92c42d63db83d2746d4100d3dd56073dbd05b98a2747bc805b43e918cf73a19e"} Oct 30 17:26:44.034871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.034802 4828 scope.go:122] "RemoveContainer" containerID="aaa0079c18457abb10c352ffe2a134f82b91761c5c205504a3f55b42f24503f4" Oct 30 17:26:44.036625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.036606 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" Oct 30 17:26:44.036737 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.036661 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d" event={"ID":"b6024600-42ca-45d6-97e5-cc0ae8071f3b","Type":"ContainerDied","Data":"3df6b1f7942ec71eac370bbbe06292e851fc34ad6c64fdb8ce133f81c182e999"} Oct 30 17:26:44.037081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.037051 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:44.038299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.038280 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:26:44.049469 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.049451 4828 scope.go:122] "RemoveContainer" containerID="aaa0079c18457abb10c352ffe2a134f82b91761c5c205504a3f55b42f24503f4" Oct 30 17:26:44.049972 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:26:44.049943 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa0079c18457abb10c352ffe2a134f82b91761c5c205504a3f55b42f24503f4\": container with ID starting with aaa0079c18457abb10c352ffe2a134f82b91761c5c205504a3f55b42f24503f4 not found: ID does not exist" containerID="aaa0079c18457abb10c352ffe2a134f82b91761c5c205504a3f55b42f24503f4" Oct 30 17:26:44.050065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.049984 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa0079c18457abb10c352ffe2a134f82b91761c5c205504a3f55b42f24503f4"} err="failed to get container status \"aaa0079c18457abb10c352ffe2a134f82b91761c5c205504a3f55b42f24503f4\": rpc error: code = NotFound desc = could not find container \"aaa0079c18457abb10c352ffe2a134f82b91761c5c205504a3f55b42f24503f4\": container with ID starting with aaa0079c18457abb10c352ffe2a134f82b91761c5c205504a3f55b42f24503f4 not found: ID does not exist" Oct 30 17:26:44.050065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.050004 4828 scope.go:122] "RemoveContainer" containerID="b4d4020998dc13e277a8293003ac41f92291711ba7086bf4a3c82191d2872949" Oct 30 17:26:44.152352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.152320 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d"] Oct 30 17:26:44.159772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.159738 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-abe243-78bf974b8b-mpn2d"] Oct 30 17:26:44.221148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.221005 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd"] Oct 30 17:26:44.224811 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.224782 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c45877-openshift-default-5dfbdd5c65-8zfjd"] Oct 30 17:26:44.306980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.306576 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:44.457884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.457850 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:44.511824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.511777 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="875c10a1-3b01-46e0-a4ed-62e65c551a91" path="/var/lib/kubelet/pods/875c10a1-3b01-46e0-a4ed-62e65c551a91/volumes" Oct 30 17:26:44.512483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.512461 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6024600-42ca-45d6-97e5-cc0ae8071f3b" path="/var/lib/kubelet/pods/b6024600-42ca-45d6-97e5-cc0ae8071f3b/volumes" Oct 30 17:26:44.802466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:44.802432 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:46.067254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:46.067224 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:46.227691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:46.227652 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:47.154118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:47.152008 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:48.563123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:48.563088 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" Oct 30 17:26:48.563591 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:48.563538 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" Oct 30 17:26:48.763138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:48.763100 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:49.190107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:49.190064 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:26:49.190306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:49.190162 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:26:50.616126 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:50.616011 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:52.225423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:52.225339 4828 generic.go:356] "Generic (PLEG): container finished" podID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerID="d7f5c6fdc49ca2b5a7b221cfa8a101002f6e42e9b840b073d7c7129932959155" exitCode=0 Oct 30 17:26:52.225847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:52.225418 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" event={"ID":"7e0b2768-4823-483d-91d6-b4bd5df2b78e","Type":"ContainerDied","Data":"d7f5c6fdc49ca2b5a7b221cfa8a101002f6e42e9b840b073d7c7129932959155"} Oct 30 17:26:52.225892 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:52.225880 4828 scope.go:122] "RemoveContainer" containerID="d7f5c6fdc49ca2b5a7b221cfa8a101002f6e42e9b840b073d7c7129932959155" Oct 30 17:26:52.228110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:52.228052 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:52.228305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:52.228241 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:52.228371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:52.228339 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:26:52.453037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:52.452940 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:53.235686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:53.235649 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" event={"ID":"7e0b2768-4823-483d-91d6-b4bd5df2b78e","Type":"ContainerStarted","Data":"444800045b23640ee4cf6397fad6688befcbea4dd73f3755dbaf0e9da8b99782"} Oct 30 17:26:54.097640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:54.097546 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:54.157659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:54.157624 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:26:54.157873 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:54.157849 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 17:26:54.157953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:54.157926 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 17:26:54.550440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:54.550381 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:55.102587 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:55.102536 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:55.158279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:55.158239 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 17:26:55.158461 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:55.158314 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 17:26:55.568807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:55.568767 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:56.005980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:55.999070 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:56.158201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:56.158165 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 17:26:56.158383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:56.158229 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 17:26:56.211985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:56.206046 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:57.158085 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:57.158053 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 17:26:57.158579 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:57.158110 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 17:26:58.158208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:58.158171 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 17:26:58.158626 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:58.158228 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 17:26:58.322380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:58.322337 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:58.744233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:58.738474 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:58.919926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:58.918792 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:59.158479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:59.158428 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body= Oct 30 17:26:59.158914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:59.158500 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" Oct 30 17:26:59.494653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:59.494567 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:26:59.664007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:26:59.663968 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:00.163322 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:00.163291 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:27:00.163838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:00.163763 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:27:00.164571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:00.164549 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:27:00.502375 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:00.502283 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:01.258632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.258598 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:01.358485 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.358439 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn"] Oct 30 17:27:01.360209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.360178 4828 cpu_manager.go:414] "containerMap: removing container" podUID="875c10a1-3b01-46e0-a4ed-62e65c551a91" containerName="istio-proxy" Oct 30 17:27:01.360387 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.360206 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875c10a1-3b01-46e0-a4ed-62e65c551a91" containerName="istio-proxy" Oct 30 17:27:01.360450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.360392 4828 cpu_manager.go:414] "containerMap: removing container" podUID="b6024600-42ca-45d6-97e5-cc0ae8071f3b" containerName="payload-processing" Oct 30 17:27:01.360450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.360397 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6024600-42ca-45d6-97e5-cc0ae8071f3b" containerName="payload-processing" Oct 30 17:27:01.360566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.360555 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="875c10a1-3b01-46e0-a4ed-62e65c551a91" containerName="istio-proxy" Oct 30 17:27:01.360623 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.360568 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6024600-42ca-45d6-97e5-cc0ae8071f3b" containerName="payload-processing" Oct 30 17:27:01.363523 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.363502 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" Oct 30 17:27:01.375253 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.375222 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn"] Oct 30 17:27:01.380083 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.380051 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-75586d\"" Oct 30 17:27:01.414842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.414794 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:01.557035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.554267 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64dc903-607a-4627-92a7-74f18599c40b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn\" (UID: \"c64dc903-607a-4627-92a7-74f18599c40b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" Oct 30 17:27:01.557035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.554419 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czs2t\" (UniqueName: \"kubernetes.io/projected/c64dc903-607a-4627-92a7-74f18599c40b-kube-api-access-czs2t\") pod \"payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn\" (UID: \"c64dc903-607a-4627-92a7-74f18599c40b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" Oct 30 17:27:01.665926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.658069 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64dc903-607a-4627-92a7-74f18599c40b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn\" (UID: \"c64dc903-607a-4627-92a7-74f18599c40b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" Oct 30 17:27:01.665926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.658205 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czs2t\" (UniqueName: \"kubernetes.io/projected/c64dc903-607a-4627-92a7-74f18599c40b-kube-api-access-czs2t\") pod \"payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn\" (UID: \"c64dc903-607a-4627-92a7-74f18599c40b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" Oct 30 17:27:01.681938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.677450 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64dc903-607a-4627-92a7-74f18599c40b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn\" (UID: \"c64dc903-607a-4627-92a7-74f18599c40b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" Oct 30 17:27:01.743963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.741214 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czs2t\" (UniqueName: \"kubernetes.io/projected/c64dc903-607a-4627-92a7-74f18599c40b-kube-api-access-czs2t\") pod \"payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn\" (UID: \"c64dc903-607a-4627-92a7-74f18599c40b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" Oct 30 17:27:01.982279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:01.982180 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" Oct 30 17:27:02.297136 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:02.297103 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn"] Oct 30 17:27:02.352114 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:02.352072 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" event={"ID":"c64dc903-607a-4627-92a7-74f18599c40b","Type":"ContainerStarted","Data":"8ddf1333bb44a74cd7caede278763a68edc91c97cec84ee76c66471014fe9a86"} Oct 30 17:27:02.868239 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:02.868198 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:03.361721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:03.361682 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" event={"ID":"c64dc903-607a-4627-92a7-74f18599c40b","Type":"ContainerStarted","Data":"10b13c58d4418eeac94bc456d8adf334803974f758c03bb0ff9e9c608bead54f"} Oct 30 17:27:03.362236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:03.361740 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" Oct 30 17:27:03.408653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:03.408576 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" podStartSLOduration=2.408556463 podStartE2EDuration="2.408556463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:03.403573539 +0000 UTC m=+2401.049083720" watchObservedRunningTime="2026-10-30 17:27:03.408556463 +0000 UTC m=+2401.054066647" Oct 30 17:27:04.765846 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:04.765808 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:06.835691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:06.835650 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:06.866725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:06.866635 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:27:06.871755 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:06.871664 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:27:07.343930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:07.342880 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:07.573934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:07.572500 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:08.050927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.046962 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4"] Oct 30 17:27:08.065926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.052570 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.117931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.113368 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9thx\" (UniqueName: \"kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-kube-api-access-f9thx\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.117931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.113500 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-data\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.117931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.113567 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-certs\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.117931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.113631 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-token\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.117931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.113698 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istiod-ca-cert\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.117931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.113727 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-podinfo\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.117931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.113761 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-socket\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.117931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.113812 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-envoy\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.117931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.113851 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-credential-socket\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.117931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.115799 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-41cb25-openshift-default-dockercfg-wsmgg\"" Oct 30 17:27:08.178459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.131619 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4"] Oct 30 17:27:08.234956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.148600 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:08.234956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.215885 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istiod-ca-cert\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.234956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.217195 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-podinfo\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.234956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.217232 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-socket\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.234956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.217290 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-envoy\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.234956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.217343 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-credential-socket\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.234956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.217445 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9thx\" (UniqueName: \"kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-kube-api-access-f9thx\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.234956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.217645 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-data\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.234956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.217736 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-certs\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.234956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.217831 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-token\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.311981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.265383 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-credential-socket\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.311981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.265663 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-socket\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.311981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.265937 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-certs\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.312695 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.312666 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-data\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.352927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.338368 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istiod-ca-cert\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.391661 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:27:08.391622 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:27:08.448976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.446669 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-token\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.479174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.479141 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9thx\" (UniqueName: \"kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-kube-api-access-f9thx\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.539070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.486648 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-envoy\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.542922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.539347 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:27:08.542922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.541338 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": context deadline exceeded" Oct 30 17:27:08.542922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.541864 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:27:08.739436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.739323 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:08.778587 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.767265 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-podinfo\") pod \"e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:08.778774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:08.778744 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:09.248820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:09.248782 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4"] Oct 30 17:27:09.256980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:09.256931 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:09.257112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:09.257029 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:09.257112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:09.257080 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:09.853378 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:09.853340 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" event={"ID":"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9","Type":"ContainerStarted","Data":"458d56cc284942d9a9c51f0ec32659d5876c43e00048d9eccec47281518d91d3"} Oct 30 17:27:09.853603 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:09.853403 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" event={"ID":"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9","Type":"ContainerStarted","Data":"bb4aba34656ef35b2882e83dec178921517994f0beaf53c19df6086f40e7ee38"} Oct 30 17:27:09.919541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:09.919477 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" podStartSLOduration=2.919457254 podStartE2EDuration="2.919457254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:09.912779611 +0000 UTC m=+2407.558289793" watchObservedRunningTime="2026-10-30 17:27:09.919457254 +0000 UTC m=+2407.564967436" Oct 30 17:27:10.096937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:10.096390 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:10.779428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:10.779395 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:10.785969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:10.785943 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:10.862183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:10.862144 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:10.863780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:10.863755 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:27:11.947976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:11.946784 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:12.379980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:12.379946 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:12.532294 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:12.532261 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:14.101089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:14.101042 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:14.251417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:14.251370 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:14.622861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:14.622822 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" Oct 30 17:27:15.396254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:15.396214 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:15.597933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:15.578851 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:15.771990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:15.771943 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4"] Oct 30 17:27:15.778119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:15.778091 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" Oct 30 17:27:15.797764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:15.797724 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4"] Oct 30 17:27:15.801836 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:15.801807 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-abe243\"" Oct 30 17:27:15.802021 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:15.801895 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-abe243-dockercfg-zmf9h\"" Oct 30 17:27:15.845522 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:15.845482 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6256\" (UniqueName: \"kubernetes.io/projected/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-kube-api-access-h6256\") pod \"payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4\" (UID: \"d7a67ffe-2b3f-47f3-8c1a-386a57b32545\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" Oct 30 17:27:15.846044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:15.846017 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4\" (UID: \"d7a67ffe-2b3f-47f3-8c1a-386a57b32545\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" Oct 30 17:27:16.023085 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:16.022998 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6256\" (UniqueName: \"kubernetes.io/projected/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-kube-api-access-h6256\") pod \"payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4\" (UID: \"d7a67ffe-2b3f-47f3-8c1a-386a57b32545\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" Oct 30 17:27:16.023507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:16.023476 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4\" (UID: \"d7a67ffe-2b3f-47f3-8c1a-386a57b32545\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" Oct 30 17:27:16.024690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:16.024667 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4\" (UID: \"d7a67ffe-2b3f-47f3-8c1a-386a57b32545\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" Oct 30 17:27:16.226237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:16.226204 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6256\" (UniqueName: \"kubernetes.io/projected/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-kube-api-access-h6256\") pod \"payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4\" (UID: \"d7a67ffe-2b3f-47f3-8c1a-386a57b32545\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" Oct 30 17:27:16.397174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:16.397074 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" Oct 30 17:27:16.725434 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:16.725399 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4"] Oct 30 17:27:16.725814 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:27:16.725782 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a67ffe_2b3f_47f3_8c1a_386a57b32545.slice/crio-d8ac4e7b094f75e8368975a61ab3151fb8e3f4a6a6b31220e501b03c03b59ed4 WatchSource:0}: Error finding container d8ac4e7b094f75e8368975a61ab3151fb8e3f4a6a6b31220e501b03c03b59ed4: Status 404 returned error can't find the container with id d8ac4e7b094f75e8368975a61ab3151fb8e3f4a6a6b31220e501b03c03b59ed4 Oct 30 17:27:16.780884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:16.780845 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:16.954587 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:16.947919 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:17.036747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:17.036715 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" event={"ID":"d7a67ffe-2b3f-47f3-8c1a-386a57b32545","Type":"ContainerStarted","Data":"d8ac4e7b094f75e8368975a61ab3151fb8e3f4a6a6b31220e501b03c03b59ed4"} Oct 30 17:27:18.496707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.496673 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5"] Oct 30 17:27:18.502418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.502389 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.528159 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.528113 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-a43843-openshift-default-dockercfg-gxd57\"" Oct 30 17:27:18.535691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.535657 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5"] Oct 30 17:27:18.664192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.662275 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-data\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.664192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.662370 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-credential-socket\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.664192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.662399 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-envoy\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.664192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.662545 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl67s\" (UniqueName: \"kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-kube-api-access-xl67s\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.664192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.662591 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-certs\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.664192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.662688 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c8de1e-84a0-4f5c-9997-3393f2de5540-istiod-ca-cert\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.664192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.662782 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-podinfo\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.664192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.662815 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-socket\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.664192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.662867 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-token\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.720779 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.720740 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:18.764284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.764217 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c8de1e-84a0-4f5c-9997-3393f2de5540-istiod-ca-cert\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.764552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.764362 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-podinfo\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.764552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.764396 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-socket\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.764552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.764448 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-token\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.764701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.764558 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-data\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.764701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.764619 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-credential-socket\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.764701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.764644 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-envoy\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.764852 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.764748 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl67s\" (UniqueName: \"kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-kube-api-access-xl67s\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.764852 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.764789 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-certs\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.765433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.765404 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-data\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.765553 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.765446 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-credential-socket\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.765553 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.765510 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-certs\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.765660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.765554 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c8de1e-84a0-4f5c-9997-3393f2de5540-istiod-ca-cert\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.765718 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.765685 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-socket\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.767228 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.767205 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-podinfo\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.768063 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.768044 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-envoy\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.789094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.789063 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-token\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.790284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.789668 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl67s\" (UniqueName: \"kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-kube-api-access-xl67s\") pod \"e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:18.838425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:18.838397 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:19.066495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:19.066392 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" event={"ID":"d7a67ffe-2b3f-47f3-8c1a-386a57b32545","Type":"ContainerStarted","Data":"0ac50c50f6ecb1cd484c3a7d29f3ea94a3a6b707edf069cfe664619ef8b25c85"} Oct 30 17:27:19.066495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:19.066490 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" Oct 30 17:27:19.188235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:19.188176 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" podStartSLOduration=2.099746536 podStartE2EDuration="4.188156847s" totalImagesPullingTime="2.088410311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:19.183706258 +0000 UTC m=+2416.829216441" watchObservedRunningTime="2026-10-30 17:27:19.188156847 +0000 UTC m=+2416.833667028" Oct 30 17:27:19.226269 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:27:19.226238 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c8de1e_84a0_4f5c_9997_3393f2de5540.slice/crio-437ad09eac444e0b60b0131a08ee474f444d2dee60227bb4e8385f30645d6f2f WatchSource:0}: Error finding container 437ad09eac444e0b60b0131a08ee474f444d2dee60227bb4e8385f30645d6f2f: Status 404 returned error can't find the container with id 437ad09eac444e0b60b0131a08ee474f444d2dee60227bb4e8385f30645d6f2f Oct 30 17:27:19.227357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:19.227331 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5"] Oct 30 17:27:19.231482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:19.230917 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:19.231482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:19.230999 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:19.231482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:19.231030 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:27:20.114974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:20.112999 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" event={"ID":"16c8de1e-84a0-4f5c-9997-3393f2de5540","Type":"ContainerStarted","Data":"e967a115e07f093f344937831efe1fc580188b19f684c8d984e5a9939aa0fd2f"} Oct 30 17:27:20.114974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:20.113084 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" event={"ID":"16c8de1e-84a0-4f5c-9997-3393f2de5540","Type":"ContainerStarted","Data":"437ad09eac444e0b60b0131a08ee474f444d2dee60227bb4e8385f30645d6f2f"} Oct 30 17:27:20.196921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:20.196410 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" podStartSLOduration=2.196385063 podStartE2EDuration="2.196385063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:20.172944473 +0000 UTC m=+2417.818454653" watchObservedRunningTime="2026-10-30 17:27:20.196385063 +0000 UTC m=+2417.841895243" Oct 30 17:27:20.607119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:20.607076 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:20.839532 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:20.839490 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:20.841791 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:20.841763 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 17:27:20.841967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:20.841816 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" podUID="16c8de1e-84a0-4f5c-9997-3393f2de5540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 17:27:21.839222 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:21.839186 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 17:27:21.839611 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:21.839241 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" podUID="16c8de1e-84a0-4f5c-9997-3393f2de5540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 17:27:22.340928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:22.334361 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:22.839348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:22.839310 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 17:27:22.839739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:22.839362 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" podUID="16c8de1e-84a0-4f5c-9997-3393f2de5540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 17:27:23.839161 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:23.839121 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 17:27:23.839325 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:23.839176 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" podUID="16c8de1e-84a0-4f5c-9997-3393f2de5540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 17:27:23.944287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:23.944244 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:24.839302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:24.839268 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 17:27:24.839487 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:24.839323 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" podUID="16c8de1e-84a0-4f5c-9997-3393f2de5540" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 17:27:25.839961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:25.839851 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:25.854252 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:25.854218 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:25.856190 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:25.856162 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:25.857790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:25.857716 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:27:27.489986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:27.489946 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:27.657526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:27.657490 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:27.871668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:27.871628 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:29.332404 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:29.332371 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:29.543797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:29.543762 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:30.118466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:30.118433 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" Oct 30 17:27:31.139005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:31.138966 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:31.396176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:31.396071 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:32.844446 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:32.844401 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:34.533930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:34.533873 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:34.796608 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:34.796521 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:35.067059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:35.066973 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:35.224550 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:35.224513 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:37.051126 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:37.051083 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:37.915996 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:27:37.915959 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:27:38.887559 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:38.887520 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:39.653038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:39.652984 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:39.807790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:39.807752 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:40.474460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:40.474419 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:41.367763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:41.367723 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:41.721973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:41.721300 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:42.383518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.383481 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f"] Oct 30 17:27:42.390174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.390145 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:42.409864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.409828 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-75586d-serving-cert\"" Oct 30 17:27:42.411411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.411351 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f"] Oct 30 17:27:42.480202 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.480163 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxh2x\" (UniqueName: \"kubernetes.io/projected/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-kube-api-access-kxh2x\") pod \"maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:42.480425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.480224 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:42.480580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.480554 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:42.480686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.480669 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:42.505760 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.505708 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:42.583206 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.583167 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:42.591679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.591646 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:42.592734 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.592710 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxh2x\" (UniqueName: \"kubernetes.io/projected/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-kube-api-access-kxh2x\") pod \"maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:42.603355 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.603318 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:42.604466 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:27:42.603723 4828 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-serving-cert: secret "maas-api-e2e-shared-b-w0-75586d-serving-cert" not found Oct 30 17:27:42.604716 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:27:42.604703 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-maas-api-tls podName:9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58 nodeName:}" failed. No retries permitted until 2026-10-30 17:27:43.104678672 +0000 UTC m=+2440.750188845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-maas-api-tls") pod "maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" (UID: "9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58") : secret "maas-api-e2e-shared-b-w0-75586d-serving-cert" not found Oct 30 17:27:42.605127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.591054 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:42.639088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.638975 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxh2x\" (UniqueName: \"kubernetes.io/projected/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-kube-api-access-kxh2x\") pod \"maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:42.639411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.639386 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:42.678425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.678386 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:42.830014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:42.829979 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:43.112300 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:43.112254 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:43.114947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:43.114916 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:43.314540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:43.314457 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:43.635714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:43.635660 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f"] Oct 30 17:27:44.293795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:44.293751 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:44.515909 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:44.515859 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" event={"ID":"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58","Type":"ContainerStarted","Data":"ab525f6463e826f5b955b858a017ffff48b57bc1f5cef15413567e2c6bcb3c33"} Oct 30 17:27:44.516108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:44.515959 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" event={"ID":"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58","Type":"ContainerStarted","Data":"76c2074c6598416ced2e67d3ebe3c35faca8c5a0ce39c16095d9d9fe2eb3ccd0"} Oct 30 17:27:44.516108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:44.515992 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:44.561161 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:44.561104 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" podStartSLOduration=2.057633706 podStartE2EDuration="2.561090804s" totalImagesPullingTime="503.457098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:44.560053412 +0000 UTC m=+2442.205563591" watchObservedRunningTime="2026-10-30 17:27:44.561090804 +0000 UTC m=+2442.206600984" Oct 30 17:27:45.521518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:45.521478 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:46.079927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:46.075984 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:46.309082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:46.309042 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:46.923682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:46.923639 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:47.115470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:47.115418 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:48.064335 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:48.064292 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:49.760227 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:49.760151 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:50.534459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:50.534397 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:27:50.758869 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:50.758834 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:50.920164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:50.920068 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:51.929361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:51.929263 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:52.548268 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:52.548211 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:52.707668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:52.707620 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:53.964082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:53.964047 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:55.702459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:55.702423 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:55.870765 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:55.870724 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:57.622390 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:57.619692 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:58.175383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:58.175351 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:58.965857 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:58.965815 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:59.553289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:59.553249 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:59.706258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:59.706227 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:27:59.864542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:27:59.864442 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:01.686284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:01.684867 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:03.294675 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:03.290819 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:05.257922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:05.251132 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:06.299215 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:06.299176 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw"] Oct 30 17:28:06.338340 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:06.338299 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss"] Oct 30 17:28:06.494017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:06.493979 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:06.647355 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:06.647223 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:06.812181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:06.812143 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:06.962193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:06.962100 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:07.915216 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:28:07.915178 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:28:08.655571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:08.655531 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:10.306561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:10.306517 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:11.300085 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.300035 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" podUID="7a6a5ffb-19f9-4c90-94b9-03c372e0e51f" containerName="payload-processing" containerID="cri-o://75ddd278b68e9af92853c6e9b7588a49b4ee8cfc55053500b3cfce664bf00c68" gracePeriod=25 Oct 30 17:28:11.338669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.338628 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" podUID="b45090c4-3935-47e9-acc0-549bc57be3f3" containerName="payload-pre-processing" containerID="cri-o://c751841da1465605bb34d75f17f9cc89e0cf08c01d16f1953656fc0b6246fd8d" gracePeriod=25 Oct 30 17:28:11.685105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.684991 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" Oct 30 17:28:11.711533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.711498 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" Oct 30 17:28:11.785764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.785718 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-kube-api-access-t9qxh\" (UniqueName: \"kubernetes.io/projected/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-kube-api-access-t9qxh\") pod \"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f\" (UID: \"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f\") " Oct 30 17:28:11.786656 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.786633 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-plugins-config-volume\") pod \"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f\" (UID: \"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f\") " Oct 30 17:28:11.787280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.787165 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-plugins-config-volume" pod "7a6a5ffb-19f9-4c90-94b9-03c372e0e51f" (UID: "7a6a5ffb-19f9-4c90-94b9-03c372e0e51f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:11.787546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.787528 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45090c4-3935-47e9-acc0-549bc57be3f3-kube-api-access-clvqj\" (UniqueName: \"kubernetes.io/projected/b45090c4-3935-47e9-acc0-549bc57be3f3-kube-api-access-clvqj\") pod \"b45090c4-3935-47e9-acc0-549bc57be3f3\" (UID: \"b45090c4-3935-47e9-acc0-549bc57be3f3\") " Oct 30 17:28:11.787988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.787971 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45090c4-3935-47e9-acc0-549bc57be3f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45090c4-3935-47e9-acc0-549bc57be3f3-plugins-config-volume\") pod \"b45090c4-3935-47e9-acc0-549bc57be3f3\" (UID: \"b45090c4-3935-47e9-acc0-549bc57be3f3\") " Oct 30 17:28:11.788624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.788512 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45090c4-3935-47e9-acc0-549bc57be3f3-plugins-config-volume" pod "b45090c4-3935-47e9-acc0-549bc57be3f3" (UID: "b45090c4-3935-47e9-acc0-549bc57be3f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:11.789467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.789446 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:28:11.789590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.789577 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45090c4-3935-47e9-acc0-549bc57be3f3-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:28:11.793354 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.793314 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-kube-api-access-t9qxh" pod "7a6a5ffb-19f9-4c90-94b9-03c372e0e51f" (UID: "7a6a5ffb-19f9-4c90-94b9-03c372e0e51f"). InnerVolumeSpecName "kube-api-access-t9qxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:11.795380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.795347 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45090c4-3935-47e9-acc0-549bc57be3f3-kube-api-access-clvqj" pod "b45090c4-3935-47e9-acc0-549bc57be3f3" (UID: "b45090c4-3935-47e9-acc0-549bc57be3f3"). InnerVolumeSpecName "kube-api-access-clvqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:11.831847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.831816 4828 generic.go:356] "Generic (PLEG): container finished" podID="b45090c4-3935-47e9-acc0-549bc57be3f3" containerID="c751841da1465605bb34d75f17f9cc89e0cf08c01d16f1953656fc0b6246fd8d" exitCode=0 Oct 30 17:28:11.832029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.831863 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" Oct 30 17:28:11.832029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.831919 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" event={"ID":"b45090c4-3935-47e9-acc0-549bc57be3f3","Type":"ContainerDied","Data":"c751841da1465605bb34d75f17f9cc89e0cf08c01d16f1953656fc0b6246fd8d"} Oct 30 17:28:11.832029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.831980 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss" event={"ID":"b45090c4-3935-47e9-acc0-549bc57be3f3","Type":"ContainerDied","Data":"cd4565a33d9486d7a8a0323a80fb347fe3cb595bd0c47e8e2345b5a835ff1174"} Oct 30 17:28:11.832029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.832012 4828 scope.go:122] "RemoveContainer" containerID="c751841da1465605bb34d75f17f9cc89e0cf08c01d16f1953656fc0b6246fd8d" Oct 30 17:28:11.833838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.833813 4828 generic.go:356] "Generic (PLEG): container finished" podID="7a6a5ffb-19f9-4c90-94b9-03c372e0e51f" containerID="75ddd278b68e9af92853c6e9b7588a49b4ee8cfc55053500b3cfce664bf00c68" exitCode=0 Oct 30 17:28:11.833959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.833851 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" event={"ID":"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f","Type":"ContainerDied","Data":"75ddd278b68e9af92853c6e9b7588a49b4ee8cfc55053500b3cfce664bf00c68"} Oct 30 17:28:11.833959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.833893 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" event={"ID":"7a6a5ffb-19f9-4c90-94b9-03c372e0e51f","Type":"ContainerDied","Data":"0e011f997aac02ab266191bbff7837deebe515a5bc6570e4c0c32f9f17e0fcf5"} Oct 30 17:28:11.834044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.834020 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw" Oct 30 17:28:11.845428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.845409 4828 scope.go:122] "RemoveContainer" containerID="c751841da1465605bb34d75f17f9cc89e0cf08c01d16f1953656fc0b6246fd8d" Oct 30 17:28:11.845715 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:28:11.845697 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c751841da1465605bb34d75f17f9cc89e0cf08c01d16f1953656fc0b6246fd8d\": container with ID starting with c751841da1465605bb34d75f17f9cc89e0cf08c01d16f1953656fc0b6246fd8d not found: ID does not exist" containerID="c751841da1465605bb34d75f17f9cc89e0cf08c01d16f1953656fc0b6246fd8d" Oct 30 17:28:11.845799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.845727 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c751841da1465605bb34d75f17f9cc89e0cf08c01d16f1953656fc0b6246fd8d"} err="failed to get container status \"c751841da1465605bb34d75f17f9cc89e0cf08c01d16f1953656fc0b6246fd8d\": rpc error: code = NotFound desc = could not find container \"c751841da1465605bb34d75f17f9cc89e0cf08c01d16f1953656fc0b6246fd8d\": container with ID starting with c751841da1465605bb34d75f17f9cc89e0cf08c01d16f1953656fc0b6246fd8d not found: ID does not exist" Oct 30 17:28:11.845799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.845758 4828 scope.go:122] "RemoveContainer" containerID="75ddd278b68e9af92853c6e9b7588a49b4ee8cfc55053500b3cfce664bf00c68" Oct 30 17:28:11.856048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.856033 4828 scope.go:122] "RemoveContainer" containerID="75ddd278b68e9af92853c6e9b7588a49b4ee8cfc55053500b3cfce664bf00c68" Oct 30 17:28:11.856303 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:28:11.856282 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ddd278b68e9af92853c6e9b7588a49b4ee8cfc55053500b3cfce664bf00c68\": container with ID starting with 75ddd278b68e9af92853c6e9b7588a49b4ee8cfc55053500b3cfce664bf00c68 not found: ID does not exist" containerID="75ddd278b68e9af92853c6e9b7588a49b4ee8cfc55053500b3cfce664bf00c68" Oct 30 17:28:11.856344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.856313 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ddd278b68e9af92853c6e9b7588a49b4ee8cfc55053500b3cfce664bf00c68"} err="failed to get container status \"75ddd278b68e9af92853c6e9b7588a49b4ee8cfc55053500b3cfce664bf00c68\": rpc error: code = NotFound desc = could not find container \"75ddd278b68e9af92853c6e9b7588a49b4ee8cfc55053500b3cfce664bf00c68\": container with ID starting with 75ddd278b68e9af92853c6e9b7588a49b4ee8cfc55053500b3cfce664bf00c68 not found: ID does not exist" Oct 30 17:28:11.887622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.887553 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw"] Oct 30 17:28:11.891638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.891606 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9qxh\" (UniqueName: \"kubernetes.io/projected/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f-kube-api-access-t9qxh\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:28:11.891638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.891640 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clvqj\" (UniqueName: \"kubernetes.io/projected/b45090c4-3935-47e9-acc0-549bc57be3f3-kube-api-access-clvqj\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:28:11.893027 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.893003 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d2d985-7d9f9ccb56-zpdrw"] Oct 30 17:28:11.926941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.926892 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss"] Oct 30 17:28:11.933405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.933380 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d2d985-64b89f65c9-lxtss"] Oct 30 17:28:11.944141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:11.944115 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:12.497842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:12.497808 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a6a5ffb-19f9-4c90-94b9-03c372e0e51f" path="/var/lib/kubelet/pods/7a6a5ffb-19f9-4c90-94b9-03c372e0e51f/volumes" Oct 30 17:28:12.498467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:12.498432 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45090c4-3935-47e9-acc0-549bc57be3f3" path="/var/lib/kubelet/pods/b45090c4-3935-47e9-acc0-549bc57be3f3/volumes" Oct 30 17:28:13.942932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:13.941417 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:15.774673 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:15.774562 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:15.990804 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:15.990756 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:16.643732 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:16.643698 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:16.815516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:16.815476 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:17.048119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.048087 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2"] Oct 30 17:28:17.049212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.049177 4828 cpu_manager.go:414] "containerMap: removing container" podUID="b45090c4-3935-47e9-acc0-549bc57be3f3" containerName="payload-pre-processing" Oct 30 17:28:17.049400 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.049375 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45090c4-3935-47e9-acc0-549bc57be3f3" containerName="payload-pre-processing" Oct 30 17:28:17.049527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.049428 4828 cpu_manager.go:414] "containerMap: removing container" podUID="7a6a5ffb-19f9-4c90-94b9-03c372e0e51f" containerName="payload-processing" Oct 30 17:28:17.049527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.049438 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a6a5ffb-19f9-4c90-94b9-03c372e0e51f" containerName="payload-processing" Oct 30 17:28:17.049703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.049684 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45090c4-3935-47e9-acc0-549bc57be3f3" containerName="payload-pre-processing" Oct 30 17:28:17.049839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.049711 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a6a5ffb-19f9-4c90-94b9-03c372e0e51f" containerName="payload-processing" Oct 30 17:28:17.055258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.055235 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.088028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.087987 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8a7e4a-openshift-default-dockercfg-z96lg\"" Oct 30 17:28:17.093467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.093436 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2"] Oct 30 17:28:17.125302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.125271 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-socket\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.125484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.125315 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-credential-socket\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.125484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.125444 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62228cc1-47cb-44f0-a0f4-9136e11b9263-istiod-ca-cert\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.125565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.125524 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-certs\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.125717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.125692 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-podinfo\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.125762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.125746 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-data\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.125865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.125844 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7pq\" (UniqueName: \"kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-kube-api-access-hj7pq\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.125973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.125956 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-token\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.126029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.126002 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-envoy\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.163072 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.163022 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:17.245931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.244967 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-podinfo\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.245931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.245030 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-data\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.245931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.245098 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7pq\" (UniqueName: \"kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-kube-api-access-hj7pq\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.245931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.245168 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-token\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.245931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.245230 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-envoy\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.245931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.245263 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-socket\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.245931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.245306 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-credential-socket\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.245931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.245397 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62228cc1-47cb-44f0-a0f4-9136e11b9263-istiod-ca-cert\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.245931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.245510 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-certs\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.246495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.246400 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-data\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.265924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.247117 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-socket\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.265924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.248356 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-credential-socket\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.274968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.273403 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-certs\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.274968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.273708 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62228cc1-47cb-44f0-a0f4-9136e11b9263-istiod-ca-cert\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.299553 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.299432 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-envoy\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.332923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.323754 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-token\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.332923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.324790 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-podinfo\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.333760 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.333694 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7pq\" (UniqueName: \"kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-kube-api-access-hj7pq\") pod \"e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.353834 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.336822 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:17.389935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.389240 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:17.513497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.513463 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:17.948101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:17.948060 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:18.052683 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:28:18.052649 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62228cc1_47cb_44f0_a0f4_9136e11b9263.slice/crio-300d6f5982e41b2c30427a620083a44d0e497920edc209ea1d1ff2b7ec331ddb WatchSource:0}: Error finding container 300d6f5982e41b2c30427a620083a44d0e497920edc209ea1d1ff2b7ec331ddb: Status 404 returned error can't find the container with id 300d6f5982e41b2c30427a620083a44d0e497920edc209ea1d1ff2b7ec331ddb Oct 30 17:28:18.052822 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:18.052737 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2"] Oct 30 17:28:18.055548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:18.055516 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:18.055615 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:18.055586 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:18.055659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:18.055615 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:18.109092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:18.109034 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:18.281022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:18.272414 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:18.926216 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:18.926161 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" event={"ID":"62228cc1-47cb-44f0-a0f4-9136e11b9263","Type":"ContainerStarted","Data":"ab28cc0e98a1f87852848f975592b9fe22f0022d53eb5fcda8c58b0cbd423494"} Oct 30 17:28:18.926449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:18.926245 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" event={"ID":"62228cc1-47cb-44f0-a0f4-9136e11b9263","Type":"ContainerStarted","Data":"300d6f5982e41b2c30427a620083a44d0e497920edc209ea1d1ff2b7ec331ddb"} Oct 30 17:28:19.045932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:19.042508 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" podStartSLOduration=2.042488188 podStartE2EDuration="2.042488188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:19.004464239 +0000 UTC m=+2476.649974422" watchObservedRunningTime="2026-10-30 17:28:19.042488188 +0000 UTC m=+2476.687998370" Oct 30 17:28:19.406052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:19.406019 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:19.462925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:19.451706 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:19.998111 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:19.998077 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:20.038280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:20.038251 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:28:21.160925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:21.125213 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:21.656924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:21.655923 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:21.858927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:21.858007 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:22.031926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:22.027716 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:22.188921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:22.188439 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:23.800924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:23.800521 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:24.229452 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:24.229364 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:24.452533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:24.452489 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:24.457620 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:24.454086 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:24.519963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:24.517531 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:24.529639 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:24.529598 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:25.655413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:25.655360 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:25.656649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:25.656625 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:28:26.054990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.054954 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf"] Oct 30 17:28:26.099929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.087347 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:26.135167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.129480 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" Oct 30 17:28:26.156261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.156224 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf"] Oct 30 17:28:26.166957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.157777 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8a7e4a\"" Oct 30 17:28:26.232771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.232719 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxrd4\" (UniqueName: \"kubernetes.io/projected/9cc646df-3fc8-4623-a773-38dd89b6763b-kube-api-access-zxrd4\") pod \"payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf\" (UID: \"9cc646df-3fc8-4623-a773-38dd89b6763b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" Oct 30 17:28:26.233116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.233097 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc646df-3fc8-4623-a773-38dd89b6763b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf\" (UID: \"9cc646df-3fc8-4623-a773-38dd89b6763b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" Oct 30 17:28:26.281152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.281024 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:26.340968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.339662 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxrd4\" (UniqueName: \"kubernetes.io/projected/9cc646df-3fc8-4623-a773-38dd89b6763b-kube-api-access-zxrd4\") pod \"payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf\" (UID: \"9cc646df-3fc8-4623-a773-38dd89b6763b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" Oct 30 17:28:26.340968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.339740 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc646df-3fc8-4623-a773-38dd89b6763b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf\" (UID: \"9cc646df-3fc8-4623-a773-38dd89b6763b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" Oct 30 17:28:26.340968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.340871 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc646df-3fc8-4623-a773-38dd89b6763b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf\" (UID: \"9cc646df-3fc8-4623-a773-38dd89b6763b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" Oct 30 17:28:26.424926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.398358 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxrd4\" (UniqueName: \"kubernetes.io/projected/9cc646df-3fc8-4623-a773-38dd89b6763b-kube-api-access-zxrd4\") pod \"payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf\" (UID: \"9cc646df-3fc8-4623-a773-38dd89b6763b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" Oct 30 17:28:26.461065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.461014 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": context deadline exceeded" Oct 30 17:28:26.516699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.516659 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:26.538314 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.534610 4828 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:26.700987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.662611 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:26.700987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.682433 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" Oct 30 17:28:26.854141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:26.832818 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk"] Oct 30 17:28:27.010934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.009712 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:27.052926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.052397 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.114924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.081926 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-63e60d-serving-cert\"" Oct 30 17:28:27.177923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.176606 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk"] Oct 30 17:28:27.216973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.216478 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:27.274977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.266975 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-63e60d-584464454b-gntmk\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.274977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.267072 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbbp7\" (UniqueName: \"kubernetes.io/projected/af3275f8-272d-40b0-84e1-d3033d1697e4-kube-api-access-pbbp7\") pod \"maas-api-e2e-ipp-b-63e60d-584464454b-gntmk\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.274977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.267133 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-metrics-certs\") pod \"maas-api-e2e-ipp-b-63e60d-584464454b-gntmk\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.274977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.267401 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3275f8-272d-40b0-84e1-d3033d1697e4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-63e60d-584464454b-gntmk\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.391938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.390504 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3275f8-272d-40b0-84e1-d3033d1697e4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-63e60d-584464454b-gntmk\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.391938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.390651 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-63e60d-584464454b-gntmk\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.391938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.390718 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbbp7\" (UniqueName: \"kubernetes.io/projected/af3275f8-272d-40b0-84e1-d3033d1697e4-kube-api-access-pbbp7\") pod \"maas-api-e2e-ipp-b-63e60d-584464454b-gntmk\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.391938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.390775 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-metrics-certs\") pod \"maas-api-e2e-ipp-b-63e60d-584464454b-gntmk\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.402351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.395858 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3275f8-272d-40b0-84e1-d3033d1697e4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-63e60d-584464454b-gntmk\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.455027 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.454988 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbbp7\" (UniqueName: \"kubernetes.io/projected/af3275f8-272d-40b0-84e1-d3033d1697e4-kube-api-access-pbbp7\") pod \"maas-api-e2e-ipp-b-63e60d-584464454b-gntmk\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.464237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.464187 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:27.466022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.465891 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:28:27.468022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.468001 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:28:27.492922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.480507 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-63e60d-584464454b-gntmk\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.516923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.500183 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-metrics-certs\") pod \"maas-api-e2e-ipp-b-63e60d-584464454b-gntmk\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.547977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.544009 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:28:27.615011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.573212 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:27.959047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:27.958564 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf"] Oct 30 17:28:27.966683 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:28:27.962720 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc646df_3fc8_4623_a773_38dd89b6763b.slice/crio-4e5772adc52d38324d4a5c06161bfabd7eef0c91e41e1c654832117663b17076 WatchSource:0}: Error finding container 4e5772adc52d38324d4a5c06161bfabd7eef0c91e41e1c654832117663b17076: Status 404 returned error can't find the container with id 4e5772adc52d38324d4a5c06161bfabd7eef0c91e41e1c654832117663b17076 Oct 30 17:28:28.290953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:28.271220 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" event={"ID":"9cc646df-3fc8-4623-a773-38dd89b6763b","Type":"ContainerStarted","Data":"66d78944ddfc32fb7efc5b42b31a32d1ca2a42ee5064cbfdf6c45456541ca997"} Oct 30 17:28:28.290953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:28.271286 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" event={"ID":"9cc646df-3fc8-4623-a773-38dd89b6763b","Type":"ContainerStarted","Data":"4e5772adc52d38324d4a5c06161bfabd7eef0c91e41e1c654832117663b17076"} Oct 30 17:28:28.290953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:28.271312 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" Oct 30 17:28:28.334313 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:28.334182 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk"] Oct 30 17:28:28.336504 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:28.336449 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" podStartSLOduration=3.336434378 podStartE2EDuration="3.336434378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:28.32767902 +0000 UTC m=+2485.973189225" watchObservedRunningTime="2026-10-30 17:28:28.336434378 +0000 UTC m=+2485.981944558" Oct 30 17:28:29.175126 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:29.175076 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:29.245517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:29.245471 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" event={"ID":"af3275f8-272d-40b0-84e1-d3033d1697e4","Type":"ContainerStarted","Data":"421bf53d4ed112be66d36b3bebb311f8f7632b19eddc42b0424fc21bf16c35af"} Oct 30 17:28:29.245757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:29.245537 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" event={"ID":"af3275f8-272d-40b0-84e1-d3033d1697e4","Type":"ContainerStarted","Data":"865aa9aa493d4451e53c6d229e51d672c114d8fe646b554bdbb7fca5b3a1d5e1"} Oct 30 17:28:29.245757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:29.245649 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:29.293776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:29.293688 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" podStartSLOduration=2.869526433 podStartE2EDuration="3.293667717s" totalImagesPullingTime="424.141284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:29.291398365 +0000 UTC m=+2486.936908570" watchObservedRunningTime="2026-10-30 17:28:29.293667717 +0000 UTC m=+2486.939177898" Oct 30 17:28:30.080105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:30.080054 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:28:30.080291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:30.080126 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:31.040925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:31.039090 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:32.720942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:32.720420 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:33.533088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:33.533053 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:28:33.553163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:33.533127 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:34.720060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:34.720015 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:35.333160 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.333128 4828 generic.go:356] "Generic (PLEG): container finished" podID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerID="1f94ee67b7770ee0a8c0c2c951a5f0942a9e18b4e64dad5ee154f2d34dd144f0" exitCode=0 Oct 30 17:28:35.333377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.333216 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" event={"ID":"f3f06243-2a82-42f6-aeaa-a187b71b4dab","Type":"ContainerDied","Data":"1f94ee67b7770ee0a8c0c2c951a5f0942a9e18b4e64dad5ee154f2d34dd144f0"} Oct 30 17:28:35.333830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.333799 4828 scope.go:122] "RemoveContainer" containerID="1f94ee67b7770ee0a8c0c2c951a5f0942a9e18b4e64dad5ee154f2d34dd144f0" Oct 30 17:28:35.335176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.335157 4828 generic.go:356] "Generic (PLEG): container finished" podID="54e68535-8263-4634-9c81-174743d3de5e" containerID="f08005d96cef9ec2453eaa4d9fc5db754dc1e35cde3d67e4fc33e6adf487f8bb" exitCode=0 Oct 30 17:28:35.335268 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.335233 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" event={"ID":"54e68535-8263-4634-9c81-174743d3de5e","Type":"ContainerDied","Data":"f08005d96cef9ec2453eaa4d9fc5db754dc1e35cde3d67e4fc33e6adf487f8bb"} Oct 30 17:28:35.335643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.335625 4828 scope.go:122] "RemoveContainer" containerID="f08005d96cef9ec2453eaa4d9fc5db754dc1e35cde3d67e4fc33e6adf487f8bb" Oct 30 17:28:35.336082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.336051 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:35.336191 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.336130 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:35.336191 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.336176 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:35.337182 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.337162 4828 generic.go:356] "Generic (PLEG): container finished" podID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerID="458d56cc284942d9a9c51f0ec32659d5876c43e00048d9eccec47281518d91d3" exitCode=0 Oct 30 17:28:35.337258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.337189 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" event={"ID":"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9","Type":"ContainerDied","Data":"458d56cc284942d9a9c51f0ec32659d5876c43e00048d9eccec47281518d91d3"} Oct 30 17:28:35.337488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.337474 4828 scope.go:122] "RemoveContainer" containerID="458d56cc284942d9a9c51f0ec32659d5876c43e00048d9eccec47281518d91d3" Oct 30 17:28:35.337534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.337475 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:35.337574 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.337534 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:35.337574 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.337563 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:35.340245 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.340212 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:35.340353 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.340294 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:35.340353 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:35.340341 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:28:36.277811 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:36.277779 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:28:36.352131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:36.352090 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" event={"ID":"f3f06243-2a82-42f6-aeaa-a187b71b4dab","Type":"ContainerStarted","Data":"072c9e8f1dcdee248f6e30898e5cf376293bc831b809f366e0a0b3843830a76a"} Oct 30 17:28:36.354962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:36.354932 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" event={"ID":"54e68535-8263-4634-9c81-174743d3de5e","Type":"ContainerStarted","Data":"ef4f971c5f31624d3fcee2077cf6e99c11d13ab3225ca55d4de7398b07d0ff50"} Oct 30 17:28:36.357842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:36.357789 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" event={"ID":"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9","Type":"ContainerStarted","Data":"370b6afdec1d5c13505e239690ae5899fb4404b6c66d1fb5a0b48957ca7c4eb9"} Oct 30 17:28:36.646852 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:36.646754 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:36.789925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:36.783841 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:28:36.789925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:36.788021 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 17:28:36.789925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:36.788093 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 17:28:36.920977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:36.920592 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:37.048978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:37.046311 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:28:37.048978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:37.046449 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:28:37.048978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:37.046497 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:28:37.339372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:37.339334 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:28:37.340137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:37.340044 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 17:28:37.340364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:37.340341 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 17:28:37.784108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:37.779722 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 17:28:37.784108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:37.779796 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 17:28:38.102003 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:28:38.080761 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:28:38.102003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:38.081508 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:28:38.102003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:38.081556 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:28:38.342409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:38.342364 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 17:28:38.343160 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:38.343122 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 17:28:38.795000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:38.794311 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 17:28:38.795000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:38.794393 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 17:28:39.104982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:39.046543 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:28:39.104982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:39.046620 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:28:39.350377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:39.350338 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 17:28:39.351355 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:39.351319 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 17:28:39.381137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:39.366916 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:39.381137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:39.376107 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" Oct 30 17:28:40.025924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:40.022103 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:40.046579 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:40.046526 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 17:28:40.046854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:40.046826 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 17:28:40.118931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:40.118551 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:28:40.118931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:40.118624 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:28:40.351770 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:40.351667 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 17:28:40.352422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:40.352388 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 17:28:40.462953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:40.461055 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:40.554500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:40.554450 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 17:28:40.834950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:40.827654 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 17:28:40.834950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:40.827717 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 17:28:41.061162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:41.061122 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:28:41.061511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:41.061459 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:28:41.117087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:41.116991 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:28:41.334933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:41.334404 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 17:28:41.334933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:41.334483 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 17:28:41.795922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:41.781670 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 17:28:41.795922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:41.781744 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 17:28:42.054747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.053559 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:28:42.054747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.053617 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:28:42.068197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.068164 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z"] Oct 30 17:28:42.196611 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.196520 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:42.312975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.293828 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:42.333052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.333014 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z"] Oct 30 17:28:42.361554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.361504 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c17e8c-serving-cert\"" Oct 30 17:28:42.361881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.361856 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 17:28:42.362062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.362036 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 17:28:42.373211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.373164 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:42.437268 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.428404 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-metrics-certs\") pod \"maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:42.437788 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.437751 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:42.449960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.438030 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg8wr\" (UniqueName: \"kubernetes.io/projected/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-kube-api-access-gg8wr\") pod \"maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:42.450405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.450376 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:42.527802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.527748 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:42.632972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.580924 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:42.632972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.582288 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:42.632972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.582463 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-metrics-certs\") pod \"maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:42.632972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.582632 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:42.632972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.582706 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg8wr\" (UniqueName: \"kubernetes.io/projected/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-kube-api-access-gg8wr\") pod \"maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:42.632972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.619257 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:42.632972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.631470 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:42.648525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.648431 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:42.676037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.659074 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg8wr\" (UniqueName: \"kubernetes.io/projected/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-kube-api-access-gg8wr\") pod \"maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:42.738701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.738663 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-metrics-certs\") pod \"maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:42.805309 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.805262 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Oct 30 17:28:42.818935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:42.805925 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Oct 30 17:28:43.024569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:43.023232 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:43.048361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:43.048321 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 30 17:28:43.048520 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:43.048385 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 30 17:28:43.332356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:43.332133 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body= Oct 30 17:28:43.332356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:43.332199 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" Oct 30 17:28:43.406346 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:28:43.406278 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2c6df0_ed33_4fff_9482_6e3cb6f6071f.slice/crio-eb8206ff09100e196e62d80b8bfba2ca5f91aa1cf0a2fd079a751256e3c2dc00 WatchSource:0}: Error finding container eb8206ff09100e196e62d80b8bfba2ca5f91aa1cf0a2fd079a751256e3c2dc00: Status 404 returned error can't find the container with id eb8206ff09100e196e62d80b8bfba2ca5f91aa1cf0a2fd079a751256e3c2dc00 Oct 30 17:28:43.410111 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:43.410081 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z"] Oct 30 17:28:43.785623 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:43.785591 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:28:43.787119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:43.787093 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:28:44.037529 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:44.037441 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:28:44.059555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:44.059514 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:28:44.138217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:44.138175 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" event={"ID":"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f","Type":"ContainerStarted","Data":"8c58aee7b3bbeb5a05c77d233d233bf10a98fc3e49a63cc13670d5370f5ae496"} Oct 30 17:28:44.138418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:44.138241 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" event={"ID":"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f","Type":"ContainerStarted","Data":"eb8206ff09100e196e62d80b8bfba2ca5f91aa1cf0a2fd079a751256e3c2dc00"} Oct 30 17:28:44.138418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:44.138285 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:44.139596 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:44.139573 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:28:44.189684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:44.189627 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" podStartSLOduration=2.824449514 podStartE2EDuration="3.189611104s" totalImagesPullingTime="365.16159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:44.185359283 +0000 UTC m=+2501.830869462" watchObservedRunningTime="2026-10-30 17:28:44.189611104 +0000 UTC m=+2501.835121321" Oct 30 17:28:44.288810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:44.288707 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:44.336477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:44.336446 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:28:44.336990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:44.336974 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:28:44.337818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:44.337794 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:28:45.087946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:45.087872 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:46.110957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:46.110919 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:47.931801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:47.931761 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:49.524162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:49.524112 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:50.152625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:50.152591 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:28:51.095760 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:51.095679 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:53.096883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:53.096844 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:55.051746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:55.051679 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:55.250335 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:55.249389 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:55.396969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:55.396866 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:55.648973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:55.647955 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:56.985367 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:56.985315 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:57.182210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:57.182146 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:58.147699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:58.147547 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:59.090195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:59.090153 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:28:59.845406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:28:59.845374 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:00.094662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:00.094577 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:01.821623 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:01.819268 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:03.586758 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:03.581360 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:05.218648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:05.218608 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:07.152924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:07.146577 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:07.911435 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:29:07.911397 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:29:09.268765 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:09.268729 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:09.798526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:09.798486 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:11.074730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:11.074677 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:11.444598 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:11.444505 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:12.761921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:12.756972 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:14.408520 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:14.408479 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:14.580658 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:14.580624 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:14.739387 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:14.739294 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:14.908695 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:14.908660 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:15.195918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.195876 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:15.221342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.221302 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672"] Oct 30 17:29:15.226346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.226318 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.242593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.242284 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-93e020-openshift-default-dockercfg-cr6fz\"" Oct 30 17:29:15.269357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.269317 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672"] Oct 30 17:29:15.383759 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.383703 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:15.417705 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.417651 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914537dc-831c-4b47-99fe-ce17f41b6247-istiod-ca-cert\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.418369 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.418345 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-socket\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.418583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.418567 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-istio-token\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.418701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.418674 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-data\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.418866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.418849 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qkp9\" (UniqueName: \"kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-kube-api-access-2qkp9\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.419125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.419109 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914537dc-831c-4b47-99fe-ce17f41b6247-istio-podinfo\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.419232 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.419219 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-credential-socket\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.419328 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.419315 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-certs\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.419490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.419446 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-envoy\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.522965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.520861 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914537dc-831c-4b47-99fe-ce17f41b6247-istiod-ca-cert\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.522965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.520946 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-socket\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.522965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.521060 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-istio-token\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.522965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.521085 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-data\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.522965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.521120 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qkp9\" (UniqueName: \"kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-kube-api-access-2qkp9\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.522965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.521249 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914537dc-831c-4b47-99fe-ce17f41b6247-istio-podinfo\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.522965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.521274 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-credential-socket\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.522965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.521300 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-certs\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.522965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.521341 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-envoy\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.532932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.530817 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-socket\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.532932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.531647 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-data\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.532932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.532139 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-credential-socket\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.536935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.535959 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-certs\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.559967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.557005 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914537dc-831c-4b47-99fe-ce17f41b6247-istiod-ca-cert\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.571248 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.571214 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-istio-token\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.588884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.586017 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914537dc-831c-4b47-99fe-ce17f41b6247-istio-podinfo\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.600988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.596783 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qkp9\" (UniqueName: \"kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-kube-api-access-2qkp9\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.647610 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.647563 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:15.650211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.650182 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-envoy\") pod \"e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:15.860978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:15.854597 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:16.310072 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:29:16.310035 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914537dc_831c_4b47_99fe_ce17f41b6247.slice/crio-ea177efb2ea10e455f1328ad45a57740f98e0c51a666cc6f50e72323f3cf6c0d WatchSource:0}: Error finding container ea177efb2ea10e455f1328ad45a57740f98e0c51a666cc6f50e72323f3cf6c0d: Status 404 returned error can't find the container with id ea177efb2ea10e455f1328ad45a57740f98e0c51a666cc6f50e72323f3cf6c0d Oct 30 17:29:16.311436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:16.311408 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672"] Oct 30 17:29:16.313575 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:16.313539 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:16.313657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:16.313607 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:16.313657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:16.313638 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:16.571604 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:16.571520 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:16.662430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:16.662391 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" event={"ID":"914537dc-831c-4b47-99fe-ce17f41b6247","Type":"ContainerStarted","Data":"a2556c4456fbb593285cfa3478bd4921e709399b03d161cbbb0580315e3f446f"} Oct 30 17:29:16.662624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:16.662458 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" event={"ID":"914537dc-831c-4b47-99fe-ce17f41b6247","Type":"ContainerStarted","Data":"ea177efb2ea10e455f1328ad45a57740f98e0c51a666cc6f50e72323f3cf6c0d"} Oct 30 17:29:16.741924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:16.739342 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:16.752926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:16.748804 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" podStartSLOduration=1.748783204 podStartE2EDuration="1.748783204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:16.718237381 +0000 UTC m=+2534.363747584" watchObservedRunningTime="2026-10-30 17:29:16.748783204 +0000 UTC m=+2534.394293385" Oct 30 17:29:16.857281 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:16.857192 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:17.864983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:17.864949 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:18.395856 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:18.395818 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:18.682208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:18.682121 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:18.683123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:18.683103 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:29:19.281599 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:19.281569 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:19.428023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:19.427986 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:20.513921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:20.513866 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:21.090373 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:21.090320 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:21.263617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:21.263580 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:22.356939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:22.352606 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:24.329506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:24.329467 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:26.083932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:26.075984 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:27.464160 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:27.464129 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:27.658810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:27.658775 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:27.815761 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:27.815720 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:28.000377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:28.000334 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:29.688151 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:29.688111 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:29.943877 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:29.943788 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk"] Oct 30 17:29:29.950095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:29.950069 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:29.980089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:29.980042 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk"] Oct 30 17:29:29.983271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:29.983241 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-469ad7-openshift-default-dockercfg-q2qkg\"" Oct 30 17:29:30.038109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.038065 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-socket\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.038317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.038229 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-data\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.038317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.038266 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-envoy\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.038508 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.038479 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istiod-ca-cert\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.038579 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.038522 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qtpf\" (UniqueName: \"kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-kube-api-access-5qtpf\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.038638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.038604 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-credential-socket\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.038700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.038660 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-token\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.038759 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.038710 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-certs\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.038822 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.038775 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-podinfo\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.140153 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.140114 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-data\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.140334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.140163 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-envoy\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.140334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.140309 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istiod-ca-cert\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.140431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.140334 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qtpf\" (UniqueName: \"kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-kube-api-access-5qtpf\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.140431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.140357 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-credential-socket\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.140431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.140388 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-token\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.140577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.140441 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-certs\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.140577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.140492 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-podinfo\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.140690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.140578 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-data\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.140690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.140605 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-socket\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.140802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.140724 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-credential-socket\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.140885 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.140850 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-certs\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.141010 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.140993 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-socket\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.141189 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.141167 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istiod-ca-cert\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.142888 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.142867 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-envoy\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.143125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.143106 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-podinfo\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.166661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.166631 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-token\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.166972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.166947 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qtpf\" (UniqueName: \"kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-kube-api-access-5qtpf\") pod \"e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.267592 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.267560 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:30.656479 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:29:30.656443 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3454fdaa_9f5d_40c0_9ce5_7a801a281024.slice/crio-22327d1fdc380c550021999db5f22355049491fb2be61fd3ac3faec44bf3a40a WatchSource:0}: Error finding container 22327d1fdc380c550021999db5f22355049491fb2be61fd3ac3faec44bf3a40a: Status 404 returned error can't find the container with id 22327d1fdc380c550021999db5f22355049491fb2be61fd3ac3faec44bf3a40a Oct 30 17:29:30.658850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.658822 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk"] Oct 30 17:29:30.659495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.659450 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:30.659616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.659530 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:30.659616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:30.659563 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:29:31.004402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:31.004297 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" event={"ID":"3454fdaa-9f5d-40c0-9ce5-7a801a281024","Type":"ContainerStarted","Data":"61eb605b9637101043a54e0e6301205b6315ad66004a216264f0f0143981a388"} Oct 30 17:29:31.004402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:31.004369 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" event={"ID":"3454fdaa-9f5d-40c0-9ce5-7a801a281024","Type":"ContainerStarted","Data":"22327d1fdc380c550021999db5f22355049491fb2be61fd3ac3faec44bf3a40a"} Oct 30 17:29:31.063005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:31.062949 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" podStartSLOduration=2.062933776 podStartE2EDuration="2.062933776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:29:31.058468566 +0000 UTC m=+2548.703978763" watchObservedRunningTime="2026-10-30 17:29:31.062933776 +0000 UTC m=+2548.708443956" Oct 30 17:29:31.268102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:31.268063 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:31.269747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:31.269708 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:29:31.269878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:31.269784 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" podUID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:29:31.470981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:31.470006 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:32.268821 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:32.268789 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:29:32.269391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:32.268839 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" podUID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:29:33.189923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:33.186528 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:33.274928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:33.273297 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:29:33.274928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:33.273361 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" podUID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:29:34.268563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:34.268529 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:29:34.268750 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:34.268582 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" podUID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:29:35.103108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:35.103055 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:35.268231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:35.268197 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:29:35.268436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:35.268258 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" podUID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:29:36.267919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:36.267868 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:29:36.268326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:36.267949 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" podUID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:29:37.280928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:37.275049 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:29:37.280928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:37.275131 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" podUID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:29:37.465597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:37.465561 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:38.172227 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:29:38.172159 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:29:38.268342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:38.268304 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 30 17:29:38.268526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:38.268378 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" podUID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 30 17:29:38.621604 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:38.621574 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:38.783480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:38.783442 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:38.968327 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:38.968244 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:39.364723 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:39.364690 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:39.366783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:39.366758 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:39.368225 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:39.368203 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:29:39.894924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:39.891705 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:41.676930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:41.673654 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:43.420208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:43.420167 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:44.617839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:44.617799 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:44.826749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:44.826710 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:45.773023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:45.772984 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:47.630138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:47.630095 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:49.785165 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:49.785073 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:51.863269 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:51.863220 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:53.597397 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:53.597358 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:55.208550 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:55.208511 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:56.772631 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:56.772582 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:58.213349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:58.213300 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:58.541927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:58.541432 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:58.907122 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:58.906997 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:59.138964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:59.131785 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:29:59.377882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:29:59.377851 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:00.339237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.339200 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:00.351073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.351035 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2"] Oct 30 17:30:00.356994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.356965 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" Oct 30 17:30:00.363029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.362983 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg"] Oct 30 17:30:00.375194 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.375131 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:30:00.375383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.375197 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2"] Oct 30 17:30:00.375383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.375307 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" Oct 30 17:30:00.380389 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.380358 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg"] Oct 30 17:30:00.499427 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.499394 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qjz\" (UniqueName: \"kubernetes.io/projected/16912205-a71d-4db8-ad39-0942880f7be6-kube-api-access-47qjz\") pod \"maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg\" (UID: \"16912205-a71d-4db8-ad39-0942880f7be6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" Oct 30 17:30:00.499640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.499516 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tt25\" (UniqueName: \"kubernetes.io/projected/63d6b63f-fb07-41bc-8462-03975bef1721-kube-api-access-8tt25\") pod \"maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2\" (UID: \"63d6b63f-fb07-41bc-8462-03975bef1721\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" Oct 30 17:30:00.499780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.499713 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16912205-a71d-4db8-ad39-0942880f7be6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg\" (UID: \"16912205-a71d-4db8-ad39-0942880f7be6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" Oct 30 17:30:00.499858 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.499842 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d6b63f-fb07-41bc-8462-03975bef1721-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2\" (UID: \"63d6b63f-fb07-41bc-8462-03975bef1721\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" Oct 30 17:30:00.601326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.601234 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tt25\" (UniqueName: \"kubernetes.io/projected/63d6b63f-fb07-41bc-8462-03975bef1721-kube-api-access-8tt25\") pod \"maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2\" (UID: \"63d6b63f-fb07-41bc-8462-03975bef1721\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" Oct 30 17:30:00.601507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.601394 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16912205-a71d-4db8-ad39-0942880f7be6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg\" (UID: \"16912205-a71d-4db8-ad39-0942880f7be6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" Oct 30 17:30:00.601591 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.601502 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d6b63f-fb07-41bc-8462-03975bef1721-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2\" (UID: \"63d6b63f-fb07-41bc-8462-03975bef1721\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" Oct 30 17:30:00.601662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.601644 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qjz\" (UniqueName: \"kubernetes.io/projected/16912205-a71d-4db8-ad39-0942880f7be6-kube-api-access-47qjz\") pod \"maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg\" (UID: \"16912205-a71d-4db8-ad39-0942880f7be6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" Oct 30 17:30:00.602190 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.602168 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d6b63f-fb07-41bc-8462-03975bef1721-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2\" (UID: \"63d6b63f-fb07-41bc-8462-03975bef1721\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" Oct 30 17:30:00.602274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.602170 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16912205-a71d-4db8-ad39-0942880f7be6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg\" (UID: \"16912205-a71d-4db8-ad39-0942880f7be6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" Oct 30 17:30:00.625608 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.625574 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qjz\" (UniqueName: \"kubernetes.io/projected/16912205-a71d-4db8-ad39-0942880f7be6-kube-api-access-47qjz\") pod \"maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg\" (UID: \"16912205-a71d-4db8-ad39-0942880f7be6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" Oct 30 17:30:00.625608 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.625576 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tt25\" (UniqueName: \"kubernetes.io/projected/63d6b63f-fb07-41bc-8462-03975bef1721-kube-api-access-8tt25\") pod \"maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2\" (UID: \"63d6b63f-fb07-41bc-8462-03975bef1721\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" Oct 30 17:30:00.695394 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.694111 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" Oct 30 17:30:00.739145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:00.719172 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" Oct 30 17:30:01.091322 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:30:01.091284 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16912205_a71d_4db8_ad39_0942880f7be6.slice/crio-1131e4317b00913e5eac4da7fe8467ab4fe42dba84495c5f0de56045144c2c9e WatchSource:0}: Error finding container 1131e4317b00913e5eac4da7fe8467ab4fe42dba84495c5f0de56045144c2c9e: Status 404 returned error can't find the container with id 1131e4317b00913e5eac4da7fe8467ab4fe42dba84495c5f0de56045144c2c9e Oct 30 17:30:01.095203 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:01.095178 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg"] Oct 30 17:30:01.104063 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:01.104034 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2"] Oct 30 17:30:01.106356 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:30:01.106328 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d6b63f_fb07_41bc_8462_03975bef1721.slice/crio-7937bff4b24ad6e53649a91c203edfe91bb90ef5d08dfb56015e3841c5cdd92d WatchSource:0}: Error finding container 7937bff4b24ad6e53649a91c203edfe91bb90ef5d08dfb56015e3841c5cdd92d: Status 404 returned error can't find the container with id 7937bff4b24ad6e53649a91c203edfe91bb90ef5d08dfb56015e3841c5cdd92d Oct 30 17:30:01.530457 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:01.530415 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" event={"ID":"63d6b63f-fb07-41bc-8462-03975bef1721","Type":"ContainerStarted","Data":"7937bff4b24ad6e53649a91c203edfe91bb90ef5d08dfb56015e3841c5cdd92d"} Oct 30 17:30:01.532129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:01.532104 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" event={"ID":"16912205-a71d-4db8-ad39-0942880f7be6","Type":"ContainerStarted","Data":"1131e4317b00913e5eac4da7fe8467ab4fe42dba84495c5f0de56045144c2c9e"} Oct 30 17:30:01.975635 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:01.975585 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:02.543872 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:02.543830 4828 generic.go:356] "Generic (PLEG): container finished" podID="63d6b63f-fb07-41bc-8462-03975bef1721" containerID="a98ef49df5299fe8aa0958c87fcc35b4ae907c315508755153593df74b516595" exitCode=60 Oct 30 17:30:02.544421 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:02.543874 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" event={"ID":"63d6b63f-fb07-41bc-8462-03975bef1721","Type":"ContainerDied","Data":"a98ef49df5299fe8aa0958c87fcc35b4ae907c315508755153593df74b516595"} Oct 30 17:30:02.544421 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:02.544346 4828 scope.go:122] "RemoveContainer" containerID="a98ef49df5299fe8aa0958c87fcc35b4ae907c315508755153593df74b516595" Oct 30 17:30:02.546101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:02.546069 4828 generic.go:356] "Generic (PLEG): container finished" podID="16912205-a71d-4db8-ad39-0942880f7be6" containerID="e119f497d3847303529d5e441c7f1a4861ad0a5a1da4b0da7ce86d981cdcdf7c" exitCode=60 Oct 30 17:30:02.546289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:02.546266 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" event={"ID":"16912205-a71d-4db8-ad39-0942880f7be6","Type":"ContainerDied","Data":"e119f497d3847303529d5e441c7f1a4861ad0a5a1da4b0da7ce86d981cdcdf7c"} Oct 30 17:30:02.546636 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:02.546538 4828 scope.go:122] "RemoveContainer" containerID="e119f497d3847303529d5e441c7f1a4861ad0a5a1da4b0da7ce86d981cdcdf7c" Oct 30 17:30:03.040608 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:03.040565 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:03.205189 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:03.205075 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:03.561501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:03.561468 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" event={"ID":"63d6b63f-fb07-41bc-8462-03975bef1721","Type":"ContainerStarted","Data":"8edb75d861d7bf7200d3e52f4d5c34303cebe77d2561b9285633d4dc9a257c39"} Oct 30 17:30:03.566368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:03.566331 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" event={"ID":"16912205-a71d-4db8-ad39-0942880f7be6","Type":"ContainerStarted","Data":"5706afdf68730b5d3dbad78c0ed1b3f03cc203489aea9722b3c705ef3d371ee2"} Oct 30 17:30:03.610942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:03.610636 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" podStartSLOduration=3.610619828 podStartE2EDuration="3.610619828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:03.609181019 +0000 UTC m=+2581.254691199" watchObservedRunningTime="2026-10-30 17:30:03.610619828 +0000 UTC m=+2581.256130010" Oct 30 17:30:03.659055 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:03.658997 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" podStartSLOduration=3.658980122 podStartE2EDuration="3.658980122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:03.655554342 +0000 UTC m=+2581.301064521" watchObservedRunningTime="2026-10-30 17:30:03.658980122 +0000 UTC m=+2581.304490302" Oct 30 17:30:03.889980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:03.886765 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:04.576990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:04.576948 4828 generic.go:356] "Generic (PLEG): container finished" podID="63d6b63f-fb07-41bc-8462-03975bef1721" containerID="8edb75d861d7bf7200d3e52f4d5c34303cebe77d2561b9285633d4dc9a257c39" exitCode=60 Oct 30 17:30:04.577509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:04.577080 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" event={"ID":"63d6b63f-fb07-41bc-8462-03975bef1721","Type":"ContainerDied","Data":"8edb75d861d7bf7200d3e52f4d5c34303cebe77d2561b9285633d4dc9a257c39"} Oct 30 17:30:04.577509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:04.577152 4828 scope.go:122] "RemoveContainer" containerID="a98ef49df5299fe8aa0958c87fcc35b4ae907c315508755153593df74b516595" Oct 30 17:30:04.577509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:04.577464 4828 scope.go:122] "RemoveContainer" containerID="8edb75d861d7bf7200d3e52f4d5c34303cebe77d2561b9285633d4dc9a257c39" Oct 30 17:30:04.578132 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:30:04.578064 4828 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2_odh-ai-gateway-infra(63d6b63f-fb07-41bc-8462-03975bef1721)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" podUID="63d6b63f-fb07-41bc-8462-03975bef1721" Oct 30 17:30:04.579511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:04.579491 4828 generic.go:356] "Generic (PLEG): container finished" podID="16912205-a71d-4db8-ad39-0942880f7be6" containerID="5706afdf68730b5d3dbad78c0ed1b3f03cc203489aea9722b3c705ef3d371ee2" exitCode=60 Oct 30 17:30:04.579610 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:04.579567 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" event={"ID":"16912205-a71d-4db8-ad39-0942880f7be6","Type":"ContainerDied","Data":"5706afdf68730b5d3dbad78c0ed1b3f03cc203489aea9722b3c705ef3d371ee2"} Oct 30 17:30:04.579981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:04.579890 4828 scope.go:122] "RemoveContainer" containerID="5706afdf68730b5d3dbad78c0ed1b3f03cc203489aea9722b3c705ef3d371ee2" Oct 30 17:30:04.580286 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:30:04.580268 4828 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg_odh-ai-gateway-infra(16912205-a71d-4db8-ad39-0942880f7be6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" podUID="16912205-a71d-4db8-ad39-0942880f7be6" Oct 30 17:30:04.594165 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:04.594139 4828 scope.go:122] "RemoveContainer" containerID="e119f497d3847303529d5e441c7f1a4861ad0a5a1da4b0da7ce86d981cdcdf7c" Oct 30 17:30:04.883118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:04.883018 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:05.010437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.010404 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc"] Oct 30 17:30:05.016693 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.016668 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" Oct 30 17:30:05.034961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.034878 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3260f8\"" Oct 30 17:30:05.035249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.035221 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3260f8-dockercfg-tt26v\"" Oct 30 17:30:05.035685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.035665 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc"] Oct 30 17:30:05.049002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.048966 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:05.314931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.262172 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49j4x\" (UniqueName: \"kubernetes.io/projected/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-kube-api-access-49j4x\") pod \"payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc\" (UID: \"9df9a223-d981-4dc6-8dfa-4ba6985b08fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" Oct 30 17:30:05.323923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.315290 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc\" (UID: \"9df9a223-d981-4dc6-8dfa-4ba6985b08fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" Oct 30 17:30:05.442920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.442725 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49j4x\" (UniqueName: \"kubernetes.io/projected/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-kube-api-access-49j4x\") pod \"payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc\" (UID: \"9df9a223-d981-4dc6-8dfa-4ba6985b08fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" Oct 30 17:30:05.442920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.442840 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc\" (UID: \"9df9a223-d981-4dc6-8dfa-4ba6985b08fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" Oct 30 17:30:05.470923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.444123 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc\" (UID: \"9df9a223-d981-4dc6-8dfa-4ba6985b08fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" Oct 30 17:30:05.470923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.470633 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49j4x\" (UniqueName: \"kubernetes.io/projected/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-kube-api-access-49j4x\") pod \"payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc\" (UID: \"9df9a223-d981-4dc6-8dfa-4ba6985b08fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" Oct 30 17:30:05.594385 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.593726 4828 scope.go:122] "RemoveContainer" containerID="8edb75d861d7bf7200d3e52f4d5c34303cebe77d2561b9285633d4dc9a257c39" Oct 30 17:30:05.594385 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:30:05.594270 4828 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2_odh-ai-gateway-infra(63d6b63f-fb07-41bc-8462-03975bef1721)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" podUID="63d6b63f-fb07-41bc-8462-03975bef1721" Oct 30 17:30:05.596009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.595988 4828 scope.go:122] "RemoveContainer" containerID="5706afdf68730b5d3dbad78c0ed1b3f03cc203489aea9722b3c705ef3d371ee2" Oct 30 17:30:05.596325 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:30:05.596303 4828 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg_odh-ai-gateway-infra(16912205-a71d-4db8-ad39-0942880f7be6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" podUID="16912205-a71d-4db8-ad39-0942880f7be6" Oct 30 17:30:05.634924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.634876 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" Oct 30 17:30:05.971053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:05.971016 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc"] Oct 30 17:30:05.971380 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:30:05.971352 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df9a223_d981_4dc6_8dfa_4ba6985b08fe.slice/crio-a61492e5a460ec8d0e668039ddf4f995997a0493121b5f5c2358a3d9a7fc2d0d WatchSource:0}: Error finding container a61492e5a460ec8d0e668039ddf4f995997a0493121b5f5c2358a3d9a7fc2d0d: Status 404 returned error can't find the container with id a61492e5a460ec8d0e668039ddf4f995997a0493121b5f5c2358a3d9a7fc2d0d Oct 30 17:30:06.081940 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:06.080750 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:06.608178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:06.608138 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" event={"ID":"9df9a223-d981-4dc6-8dfa-4ba6985b08fe","Type":"ContainerStarted","Data":"f76f6df6170281b3ed463e99a7e0c07ef4f802135ab05527ed4b22977adc0343"} Oct 30 17:30:06.608586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:06.608207 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" event={"ID":"9df9a223-d981-4dc6-8dfa-4ba6985b08fe","Type":"ContainerStarted","Data":"a61492e5a460ec8d0e668039ddf4f995997a0493121b5f5c2358a3d9a7fc2d0d"} Oct 30 17:30:06.608586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:06.608251 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" Oct 30 17:30:06.659625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:06.659571 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" podStartSLOduration=2.659555337 podStartE2EDuration="2.659555337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:06.655365871 +0000 UTC m=+2584.300876052" watchObservedRunningTime="2026-10-30 17:30:06.659555337 +0000 UTC m=+2584.305065516" Oct 30 17:30:07.792946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:07.792837 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:07.934579 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:30:07.934521 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:30:09.690924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:09.687556 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:11.433779 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.433729 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6"] Oct 30 17:30:11.440413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.440391 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:11.458201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.458169 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-93e020-serving-cert\"" Oct 30 17:30:11.469020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.468988 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6"] Oct 30 17:30:11.508430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.508390 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp2n8\" (UniqueName: \"kubernetes.io/projected/f11b518b-9d3f-4422-98a3-82b684e2881e-kube-api-access-bp2n8\") pod \"maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:11.508635 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.508609 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-metrics-certs\") pod \"maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:11.508703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.508669 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11b518b-9d3f-4422-98a3-82b684e2881e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:11.509034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.509012 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:11.551660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.551623 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:11.633924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.619533 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:11.633924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.619781 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp2n8\" (UniqueName: \"kubernetes.io/projected/f11b518b-9d3f-4422-98a3-82b684e2881e-kube-api-access-bp2n8\") pod \"maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:11.633924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.619847 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-metrics-certs\") pod \"maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:11.633924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.619879 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11b518b-9d3f-4422-98a3-82b684e2881e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:11.656950 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:30:11.637110 4828 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-serving-cert: secret "maas-api-e2e-ipp-b-93e020-serving-cert" not found Oct 30 17:30:11.656950 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:30:11.637202 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-maas-api-tls podName:f11b518b-9d3f-4422-98a3-82b684e2881e nodeName:}" failed. No retries permitted until 2026-10-30 17:30:12.137178529 +0000 UTC m=+2589.782688700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-maas-api-tls") pod "maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" (UID: "f11b518b-9d3f-4422-98a3-82b684e2881e") : secret "maas-api-e2e-ipp-b-93e020-serving-cert" not found Oct 30 17:30:11.656950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.639362 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11b518b-9d3f-4422-98a3-82b684e2881e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:11.761857 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.760810 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:11.761857 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.761540 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-metrics-certs\") pod \"maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:11.785925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.782201 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp2n8\" (UniqueName: \"kubernetes.io/projected/f11b518b-9d3f-4422-98a3-82b684e2881e-kube-api-access-bp2n8\") pod \"maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:11.922344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:11.922305 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:12.160892 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:12.160800 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:12.163353 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:12.163321 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:12.359069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:12.359028 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:12.718040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:12.718003 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6"] Oct 30 17:30:12.924953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:12.920851 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" event={"ID":"f11b518b-9d3f-4422-98a3-82b684e2881e","Type":"ContainerStarted","Data":"f4d1513eb2525c0b77d76684868992922eccdd4db14b2d6adaa8d9815f1b6114"} Oct 30 17:30:13.730041 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:13.730005 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:13.931281 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:13.931238 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" event={"ID":"f11b518b-9d3f-4422-98a3-82b684e2881e","Type":"ContainerStarted","Data":"d3c0dc41aa8d513c63c93e876584d25e92d6992d5d30e8fa498f723920532047"} Oct 30 17:30:13.931471 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:13.931340 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:14.008071 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:14.008017 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" podStartSLOduration=2.4963091029999998 podStartE2EDuration="3.007999588s" totalImagesPullingTime="511.690485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:13.992371967 +0000 UTC m=+2591.637882148" watchObservedRunningTime="2026-10-30 17:30:14.007999588 +0000 UTC m=+2591.653509808" Oct 30 17:30:14.389988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:14.389645 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:14.495924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:14.495666 4828 scope.go:122] "RemoveContainer" containerID="5706afdf68730b5d3dbad78c0ed1b3f03cc203489aea9722b3c705ef3d371ee2" Oct 30 17:30:14.520481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:14.520434 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf"] Oct 30 17:30:14.674036 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:14.673758 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:14.845992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:14.844805 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:15.022924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:15.019066 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:15.645546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:15.645510 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:16.017971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:16.017935 4828 generic.go:356] "Generic (PLEG): container finished" podID="16912205-a71d-4db8-ad39-0942880f7be6" containerID="9256b970aeecb4844dda75ecd0ce93efd3e6b231c04390238a9a8bc7cc3d45b9" exitCode=60 Oct 30 17:30:16.018490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:16.018009 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" event={"ID":"16912205-a71d-4db8-ad39-0942880f7be6","Type":"ContainerDied","Data":"9256b970aeecb4844dda75ecd0ce93efd3e6b231c04390238a9a8bc7cc3d45b9"} Oct 30 17:30:16.018490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:16.018074 4828 scope.go:122] "RemoveContainer" containerID="5706afdf68730b5d3dbad78c0ed1b3f03cc203489aea9722b3c705ef3d371ee2" Oct 30 17:30:16.018620 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:16.018532 4828 scope.go:122] "RemoveContainer" containerID="9256b970aeecb4844dda75ecd0ce93efd3e6b231c04390238a9a8bc7cc3d45b9" Oct 30 17:30:16.019001 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:30:16.018972 4828 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg_odh-ai-gateway-infra(16912205-a71d-4db8-ad39-0942880f7be6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" podUID="16912205-a71d-4db8-ad39-0942880f7be6" Oct 30 17:30:16.721746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:16.721705 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:17.153761 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:17.153710 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg"] Oct 30 17:30:17.508747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:17.508717 4828 scope.go:122] "RemoveContainer" containerID="8edb75d861d7bf7200d3e52f4d5c34303cebe77d2561b9285633d4dc9a257c39" Oct 30 17:30:17.514745 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:17.514714 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" Oct 30 17:30:17.533461 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:17.533434 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16912205-a71d-4db8-ad39-0942880f7be6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16912205-a71d-4db8-ad39-0942880f7be6-maas-api-service-ca\") pod \"16912205-a71d-4db8-ad39-0942880f7be6\" (UID: \"16912205-a71d-4db8-ad39-0942880f7be6\") " Oct 30 17:30:17.533573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:17.533540 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16912205-a71d-4db8-ad39-0942880f7be6-kube-api-access-47qjz\" (UniqueName: \"kubernetes.io/projected/16912205-a71d-4db8-ad39-0942880f7be6-kube-api-access-47qjz\") pod \"16912205-a71d-4db8-ad39-0942880f7be6\" (UID: \"16912205-a71d-4db8-ad39-0942880f7be6\") " Oct 30 17:30:17.534028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:17.534003 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16912205-a71d-4db8-ad39-0942880f7be6-maas-api-service-ca" pod "16912205-a71d-4db8-ad39-0942880f7be6" (UID: "16912205-a71d-4db8-ad39-0942880f7be6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:17.536371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:17.536350 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16912205-a71d-4db8-ad39-0942880f7be6-kube-api-access-47qjz" pod "16912205-a71d-4db8-ad39-0942880f7be6" (UID: "16912205-a71d-4db8-ad39-0942880f7be6"). InnerVolumeSpecName "kube-api-access-47qjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:17.630645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:17.630612 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" Oct 30 17:30:17.635849 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:17.635799 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16912205-a71d-4db8-ad39-0942880f7be6-maas-api-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:30:17.635849 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:17.635835 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47qjz\" (UniqueName: \"kubernetes.io/projected/16912205-a71d-4db8-ad39-0942880f7be6-kube-api-access-47qjz\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:30:17.759060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:17.759000 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:18.133481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:18.133439 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" event={"ID":"63d6b63f-fb07-41bc-8462-03975bef1721","Type":"ContainerStarted","Data":"b60e3fe094d411088e3b5f716139ac7bf1395ff4b8c3efe8a195a683b0125739"} Oct 30 17:30:18.135075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:18.135050 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" Oct 30 17:30:18.135232 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:18.135103 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg" event={"ID":"16912205-a71d-4db8-ad39-0942880f7be6","Type":"ContainerDied","Data":"1131e4317b00913e5eac4da7fe8467ab4fe42dba84495c5f0de56045144c2c9e"} Oct 30 17:30:18.135232 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:18.135159 4828 scope.go:122] "RemoveContainer" containerID="9256b970aeecb4844dda75ecd0ce93efd3e6b231c04390238a9a8bc7cc3d45b9" Oct 30 17:30:18.224149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:18.224107 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg"] Oct 30 17:30:18.233114 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:18.232754 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-41cb25-29889690-9hxwg"] Oct 30 17:30:18.496792 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:18.496698 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16912205-a71d-4db8-ad39-0942880f7be6" path="/var/lib/kubelet/pods/16912205-a71d-4db8-ad39-0942880f7be6/volumes" Oct 30 17:30:19.122801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:19.122755 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:19.148257 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:19.148227 4828 generic.go:356] "Generic (PLEG): container finished" podID="63d6b63f-fb07-41bc-8462-03975bef1721" containerID="b60e3fe094d411088e3b5f716139ac7bf1395ff4b8c3efe8a195a683b0125739" exitCode=60 Oct 30 17:30:19.148467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:19.148307 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" event={"ID":"63d6b63f-fb07-41bc-8462-03975bef1721","Type":"ContainerDied","Data":"b60e3fe094d411088e3b5f716139ac7bf1395ff4b8c3efe8a195a683b0125739"} Oct 30 17:30:19.148467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:19.148383 4828 scope.go:122] "RemoveContainer" containerID="8edb75d861d7bf7200d3e52f4d5c34303cebe77d2561b9285633d4dc9a257c39" Oct 30 17:30:19.148871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:19.148850 4828 scope.go:122] "RemoveContainer" containerID="b60e3fe094d411088e3b5f716139ac7bf1395ff4b8c3efe8a195a683b0125739" Oct 30 17:30:19.149341 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:30:19.149315 4828 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2_odh-ai-gateway-infra(63d6b63f-fb07-41bc-8462-03975bef1721)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" podUID="63d6b63f-fb07-41bc-8462-03975bef1721" Oct 30 17:30:19.384114 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:19.384024 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2"] Oct 30 17:30:19.397973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:19.397860 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:19.533981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:19.526827 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" podUID="9cc646df-3fc8-4623-a773-38dd89b6763b" containerName="payload-pre-processing" containerID="cri-o://66d78944ddfc32fb7efc5b42b31a32d1ca2a42ee5064cbfdf6c45456541ca997" gracePeriod=25 Oct 30 17:30:20.045049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.044997 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:30:20.509780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.509724 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" Oct 30 17:30:20.574997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.574001 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" Oct 30 17:30:20.574997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.574721 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d6b63f-fb07-41bc-8462-03975bef1721-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d6b63f-fb07-41bc-8462-03975bef1721-maas-api-service-ca\") pod \"63d6b63f-fb07-41bc-8462-03975bef1721\" (UID: \"63d6b63f-fb07-41bc-8462-03975bef1721\") " Oct 30 17:30:20.574997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.574790 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d6b63f-fb07-41bc-8462-03975bef1721-kube-api-access-8tt25\" (UniqueName: \"kubernetes.io/projected/63d6b63f-fb07-41bc-8462-03975bef1721-kube-api-access-8tt25\") pod \"63d6b63f-fb07-41bc-8462-03975bef1721\" (UID: \"63d6b63f-fb07-41bc-8462-03975bef1721\") " Oct 30 17:30:20.577817 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.577240 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d6b63f-fb07-41bc-8462-03975bef1721-maas-api-service-ca" pod "63d6b63f-fb07-41bc-8462-03975bef1721" (UID: "63d6b63f-fb07-41bc-8462-03975bef1721"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:20.581327 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.581288 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d6b63f-fb07-41bc-8462-03975bef1721-kube-api-access-8tt25" pod "63d6b63f-fb07-41bc-8462-03975bef1721" (UID: "63d6b63f-fb07-41bc-8462-03975bef1721"). InnerVolumeSpecName "kube-api-access-8tt25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:20.640192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.639895 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:20.688924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.688148 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc646df-3fc8-4623-a773-38dd89b6763b-kube-api-access-zxrd4\" (UniqueName: \"kubernetes.io/projected/9cc646df-3fc8-4623-a773-38dd89b6763b-kube-api-access-zxrd4\") pod \"9cc646df-3fc8-4623-a773-38dd89b6763b\" (UID: \"9cc646df-3fc8-4623-a773-38dd89b6763b\") " Oct 30 17:30:20.688924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.688241 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cc646df-3fc8-4623-a773-38dd89b6763b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc646df-3fc8-4623-a773-38dd89b6763b-plugins-config-volume\") pod \"9cc646df-3fc8-4623-a773-38dd89b6763b\" (UID: \"9cc646df-3fc8-4623-a773-38dd89b6763b\") " Oct 30 17:30:20.689195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.689145 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d6b63f-fb07-41bc-8462-03975bef1721-maas-api-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.689195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.689168 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tt25\" (UniqueName: \"kubernetes.io/projected/63d6b63f-fb07-41bc-8462-03975bef1721-kube-api-access-8tt25\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.716049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.703204 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc646df-3fc8-4623-a773-38dd89b6763b-plugins-config-volume" pod "9cc646df-3fc8-4623-a773-38dd89b6763b" (UID: "9cc646df-3fc8-4623-a773-38dd89b6763b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:20.789030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.788883 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc646df-3fc8-4623-a773-38dd89b6763b-kube-api-access-zxrd4" pod "9cc646df-3fc8-4623-a773-38dd89b6763b" (UID: "9cc646df-3fc8-4623-a773-38dd89b6763b"). InnerVolumeSpecName "kube-api-access-zxrd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:20.791812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.791065 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxrd4\" (UniqueName: \"kubernetes.io/projected/9cc646df-3fc8-4623-a773-38dd89b6763b-kube-api-access-zxrd4\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:30:20.791812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:20.791100 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cc646df-3fc8-4623-a773-38dd89b6763b-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:30:21.044059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.044023 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:21.044238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.044098 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:21.178966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.177295 4828 generic.go:356] "Generic (PLEG): container finished" podID="9cc646df-3fc8-4623-a773-38dd89b6763b" containerID="66d78944ddfc32fb7efc5b42b31a32d1ca2a42ee5064cbfdf6c45456541ca997" exitCode=0 Oct 30 17:30:21.178966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.177378 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" Oct 30 17:30:21.178966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.177383 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" event={"ID":"9cc646df-3fc8-4623-a773-38dd89b6763b","Type":"ContainerDied","Data":"66d78944ddfc32fb7efc5b42b31a32d1ca2a42ee5064cbfdf6c45456541ca997"} Oct 30 17:30:21.178966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.177485 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf" event={"ID":"9cc646df-3fc8-4623-a773-38dd89b6763b","Type":"ContainerDied","Data":"4e5772adc52d38324d4a5c06161bfabd7eef0c91e41e1c654832117663b17076"} Oct 30 17:30:21.178966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.177517 4828 scope.go:122] "RemoveContainer" containerID="66d78944ddfc32fb7efc5b42b31a32d1ca2a42ee5064cbfdf6c45456541ca997" Oct 30 17:30:21.185699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.185197 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" event={"ID":"63d6b63f-fb07-41bc-8462-03975bef1721","Type":"ContainerDied","Data":"7937bff4b24ad6e53649a91c203edfe91bb90ef5d08dfb56015e3841c5cdd92d"} Oct 30 17:30:21.185699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.185301 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2" Oct 30 17:30:21.207966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.207941 4828 scope.go:122] "RemoveContainer" containerID="66d78944ddfc32fb7efc5b42b31a32d1ca2a42ee5064cbfdf6c45456541ca997" Oct 30 17:30:21.209190 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:30:21.209145 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d78944ddfc32fb7efc5b42b31a32d1ca2a42ee5064cbfdf6c45456541ca997\": container with ID starting with 66d78944ddfc32fb7efc5b42b31a32d1ca2a42ee5064cbfdf6c45456541ca997 not found: ID does not exist" containerID="66d78944ddfc32fb7efc5b42b31a32d1ca2a42ee5064cbfdf6c45456541ca997" Oct 30 17:30:21.209330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.209202 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d78944ddfc32fb7efc5b42b31a32d1ca2a42ee5064cbfdf6c45456541ca997"} err="failed to get container status \"66d78944ddfc32fb7efc5b42b31a32d1ca2a42ee5064cbfdf6c45456541ca997\": rpc error: code = NotFound desc = could not find container \"66d78944ddfc32fb7efc5b42b31a32d1ca2a42ee5064cbfdf6c45456541ca997\": container with ID starting with 66d78944ddfc32fb7efc5b42b31a32d1ca2a42ee5064cbfdf6c45456541ca997 not found: ID does not exist" Oct 30 17:30:21.209330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.209237 4828 scope.go:122] "RemoveContainer" containerID="b60e3fe094d411088e3b5f716139ac7bf1395ff4b8c3efe8a195a683b0125739" Oct 30 17:30:21.259750 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.259713 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf"] Oct 30 17:30:21.279354 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.279317 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8a7e4a-599c987679-v5xlf"] Oct 30 17:30:21.305577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.305488 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2"] Oct 30 17:30:21.311925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:21.311862 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-abe243-29889690-z85g2"] Oct 30 17:30:22.481524 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:22.481488 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:22.547180 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:22.537803 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d6b63f-fb07-41bc-8462-03975bef1721" path="/var/lib/kubelet/pods/63d6b63f-fb07-41bc-8462-03975bef1721/volumes" Oct 30 17:30:22.548336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:22.548300 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cc646df-3fc8-4623-a773-38dd89b6763b" path="/var/lib/kubelet/pods/9cc646df-3fc8-4623-a773-38dd89b6763b/volumes" Oct 30 17:30:24.051926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:24.050621 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:24.225791 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:24.225761 4828 generic.go:356] "Generic (PLEG): container finished" podID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerID="ab28cc0e98a1f87852848f975592b9fe22f0022d53eb5fcda8c58b0cbd423494" exitCode=0 Oct 30 17:30:24.226010 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:24.225801 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" event={"ID":"62228cc1-47cb-44f0-a0f4-9136e11b9263","Type":"ContainerDied","Data":"ab28cc0e98a1f87852848f975592b9fe22f0022d53eb5fcda8c58b0cbd423494"} Oct 30 17:30:24.226369 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:24.226346 4828 scope.go:122] "RemoveContainer" containerID="ab28cc0e98a1f87852848f975592b9fe22f0022d53eb5fcda8c58b0cbd423494" Oct 30 17:30:24.228701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:24.228669 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:30:24.228807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:24.228740 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:30:24.228807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:24.228774 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:30:25.237409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:25.237370 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" event={"ID":"62228cc1-47cb-44f0-a0f4-9136e11b9263","Type":"ContainerStarted","Data":"1315ad1c6b1c90459df836d38610fdc28365b335f7458bfc08a7d3564ca15d16"} Oct 30 17:30:25.392735 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:25.392695 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:30:25.393139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:25.393038 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 17:30:25.393318 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:25.393296 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 17:30:25.777884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:25.777847 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:26.394353 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:26.394307 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 17:30:26.394789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:26.394393 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 17:30:27.393370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:27.393330 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 17:30:27.393679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:27.393650 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 17:30:27.509270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:27.509229 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:28.399927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:28.391313 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 17:30:28.399927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:28.391381 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 17:30:29.391924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:29.391335 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 17:30:29.391924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:29.391405 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 17:30:29.539393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:29.539345 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:30.391924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:30.391315 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Oct 30 17:30:30.391924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:30.391381 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Oct 30 17:30:31.333698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:31.332492 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:31.418265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:31.418229 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:30:31.420690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:31.420662 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:30:31.427185 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:31.427153 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:30:32.563845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:32.563810 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:32.782940 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:32.780227 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:33.745873 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:33.745829 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:35.470927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:35.462852 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:37.291108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:37.289440 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:37.917225 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:30:37.917184 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:30:39.268447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:39.268378 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:41.439337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:41.434715 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:43.301105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:43.301056 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:45.547688 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:45.547649 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:46.931773 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:46.931739 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:47.528975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:47.507236 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:47.761171 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:47.761114 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:48.351918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:48.351870 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:49.315091 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:49.315050 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:50.220713 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:50.220671 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:51.878080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:51.877791 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:52.036237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:52.036197 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:52.292173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:52.292127 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:53.656661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:53.656602 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:53.813578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:53.813544 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:54.200926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:54.173484 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:56.191032 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:56.190114 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:57.821924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:57.817605 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:59.478499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.478448 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4"] Oct 30 17:30:59.479583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479562 4828 cpu_manager.go:414] "containerMap: removing container" podUID="63d6b63f-fb07-41bc-8462-03975bef1721" containerName="cleanup" Oct 30 17:30:59.479583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479583 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d6b63f-fb07-41bc-8462-03975bef1721" containerName="cleanup" Oct 30 17:30:59.479772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479604 4828 cpu_manager.go:414] "containerMap: removing container" podUID="16912205-a71d-4db8-ad39-0942880f7be6" containerName="cleanup" Oct 30 17:30:59.479772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479615 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16912205-a71d-4db8-ad39-0942880f7be6" containerName="cleanup" Oct 30 17:30:59.479772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479673 4828 cpu_manager.go:414] "containerMap: removing container" podUID="63d6b63f-fb07-41bc-8462-03975bef1721" containerName="cleanup" Oct 30 17:30:59.479772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479680 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d6b63f-fb07-41bc-8462-03975bef1721" containerName="cleanup" Oct 30 17:30:59.479772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479718 4828 cpu_manager.go:414] "containerMap: removing container" podUID="9cc646df-3fc8-4623-a773-38dd89b6763b" containerName="payload-pre-processing" Oct 30 17:30:59.479772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479727 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc646df-3fc8-4623-a773-38dd89b6763b" containerName="payload-pre-processing" Oct 30 17:30:59.479772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479747 4828 cpu_manager.go:414] "containerMap: removing container" podUID="16912205-a71d-4db8-ad39-0942880f7be6" containerName="cleanup" Oct 30 17:30:59.479772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479755 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16912205-a71d-4db8-ad39-0942880f7be6" containerName="cleanup" Oct 30 17:30:59.480262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479960 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="16912205-a71d-4db8-ad39-0942880f7be6" containerName="cleanup" Oct 30 17:30:59.480262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479976 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d6b63f-fb07-41bc-8462-03975bef1721" containerName="cleanup" Oct 30 17:30:59.480262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479986 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="16912205-a71d-4db8-ad39-0942880f7be6" containerName="cleanup" Oct 30 17:30:59.480262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.479997 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="16912205-a71d-4db8-ad39-0942880f7be6" containerName="cleanup" Oct 30 17:30:59.480262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.480015 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc646df-3fc8-4623-a773-38dd89b6763b" containerName="payload-pre-processing" Oct 30 17:30:59.480262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.480022 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d6b63f-fb07-41bc-8462-03975bef1721" containerName="cleanup" Oct 30 17:30:59.485793 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.485760 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.506298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.506265 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3260f8-serving-cert\"" Oct 30 17:30:59.508022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.507986 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:59.513559 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.513528 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4"] Oct 30 17:30:59.602723 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.602683 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbgff\" (UniqueName: \"kubernetes.io/projected/1de99d4d-0549-4795-8e4b-1d1554a315c4-kube-api-access-nbgff\") pod \"maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.603055 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.603030 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de99d4d-0549-4795-8e4b-1d1554a315c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.603257 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.603231 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.603320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.603279 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-metrics-certs\") pod \"maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.678929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.665173 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:30:59.705765 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.705711 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbgff\" (UniqueName: \"kubernetes.io/projected/1de99d4d-0549-4795-8e4b-1d1554a315c4-kube-api-access-nbgff\") pod \"maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.706118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.706085 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de99d4d-0549-4795-8e4b-1d1554a315c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.707245 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.707039 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de99d4d-0549-4795-8e4b-1d1554a315c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.707370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.707321 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.707370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.707356 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-metrics-certs\") pod \"maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.735977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.730955 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.740171 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.739966 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-metrics-certs\") pod \"maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.754079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.748358 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbgff\" (UniqueName: \"kubernetes.io/projected/1de99d4d-0549-4795-8e4b-1d1554a315c4-kube-api-access-nbgff\") pod \"maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.811210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.811173 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:30:59.824832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:30:59.824700 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:00.144519 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:31:00.144480 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de99d4d_0549_4795_8e4b_1d1554a315c4.slice/crio-ea222e432e4ab576a1c34579c7910106a70c158c35c3c92401150455d90b2d5e WatchSource:0}: Error finding container ea222e432e4ab576a1c34579c7910106a70c158c35c3c92401150455d90b2d5e: Status 404 returned error can't find the container with id ea222e432e4ab576a1c34579c7910106a70c158c35c3c92401150455d90b2d5e Oct 30 17:31:00.146958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:00.146928 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4"] Oct 30 17:31:00.163291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:00.163249 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" event={"ID":"1de99d4d-0549-4795-8e4b-1d1554a315c4","Type":"ContainerStarted","Data":"ea222e432e4ab576a1c34579c7910106a70c158c35c3c92401150455d90b2d5e"} Oct 30 17:31:00.217522 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:00.217468 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:01.174023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:01.173979 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" event={"ID":"1de99d4d-0549-4795-8e4b-1d1554a315c4","Type":"ContainerStarted","Data":"68b26f7a56fcc6b2c8241cf66692dd47858deba8a6f40084c231f2abf9562b69"} Oct 30 17:31:01.224301 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:01.224245 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" podStartSLOduration=1.848343992 podStartE2EDuration="2.224231252s" totalImagesPullingTime="375.88726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:01.218197257 +0000 UTC m=+2638.863707438" watchObservedRunningTime="2026-10-30 17:31:01.224231252 +0000 UTC m=+2638.869741431" Oct 30 17:31:01.591264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:01.583317 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:02.184993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:02.184955 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:31:03.193576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:03.193523 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:04.854367 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:04.854082 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:06.582184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:06.581508 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:07.915808 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:31:07.915769 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:31:08.238262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:08.238187 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:31:08.337145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:08.337069 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:09.543380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:09.543348 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:09.766938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:09.755720 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:11.130127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:11.130092 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:13.999117 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:13.999076 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:15.986546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:15.986504 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:17.410920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:17.401634 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:17.599266 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:17.590133 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:17.785988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:17.778419 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:18.167987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:18.160973 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:20.561686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:20.561643 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:22.122585 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:22.122548 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:24.044882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:24.044843 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:24.737381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:24.737348 4828 generic.go:356] "Generic (PLEG): container finished" podID="914537dc-831c-4b47-99fe-ce17f41b6247" containerID="a2556c4456fbb593285cfa3478bd4921e709399b03d161cbbb0580315e3f446f" exitCode=0 Oct 30 17:31:24.737588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:24.737442 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" event={"ID":"914537dc-831c-4b47-99fe-ce17f41b6247","Type":"ContainerDied","Data":"a2556c4456fbb593285cfa3478bd4921e709399b03d161cbbb0580315e3f446f"} Oct 30 17:31:24.737998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:24.737978 4828 scope.go:122] "RemoveContainer" containerID="a2556c4456fbb593285cfa3478bd4921e709399b03d161cbbb0580315e3f446f" Oct 30 17:31:24.740195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:24.740167 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:31:24.740270 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:24.740242 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:31:24.740309 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:24.740291 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:31:25.716180 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:25.716142 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:25.801925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:25.794280 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" event={"ID":"914537dc-831c-4b47-99fe-ce17f41b6247","Type":"ContainerStarted","Data":"7f0b75f97d03b9c3833c8e05c4553ac6106a4f61d57a9af0099ff0f1ee0f11f0"} Oct 30 17:31:25.856781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:25.856743 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:31:25.857044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:25.857005 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:31:25.857103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:25.857060 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:31:26.409606 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.409571 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt"] Oct 30 17:31:26.410450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.410433 4828 cpu_manager.go:414] "containerMap: removing container" podUID="63d6b63f-fb07-41bc-8462-03975bef1721" containerName="cleanup" Oct 30 17:31:26.410505 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.410451 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d6b63f-fb07-41bc-8462-03975bef1721" containerName="cleanup" Oct 30 17:31:26.410505 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.410487 4828 cpu_manager.go:414] "containerMap: removing container" podUID="16912205-a71d-4db8-ad39-0942880f7be6" containerName="cleanup" Oct 30 17:31:26.410505 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.410493 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16912205-a71d-4db8-ad39-0942880f7be6" containerName="cleanup" Oct 30 17:31:26.410646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.410636 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d6b63f-fb07-41bc-8462-03975bef1721" containerName="cleanup" Oct 30 17:31:26.413939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.413884 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.422447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.422412 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt"] Oct 30 17:31:26.430691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.430364 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-75586d-openshift-default-dockercfg-kkkg2\"" Oct 30 17:31:26.561459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.561414 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-envoy\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.561639 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.561541 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-podinfo\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.561639 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.561610 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb25a1e-9f6b-4823-982a-82b25909fd53-istiod-ca-cert\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.561751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.561665 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-token\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.561751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.561700 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6phkn\" (UniqueName: \"kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-kube-api-access-6phkn\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.561910 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.561872 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-data\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.561985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.561970 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-credential-socket\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.562097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.562085 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-certs\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.562146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.562136 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-socket\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.663801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.663709 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-podinfo\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.663801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.663774 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb25a1e-9f6b-4823-982a-82b25909fd53-istiod-ca-cert\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.664044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.663825 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-token\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.664044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.663860 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6phkn\" (UniqueName: \"kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-kube-api-access-6phkn\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.664044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.663983 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-data\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.664044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.664030 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-credential-socket\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.664284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.664193 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-certs\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.664284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.664274 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-socket\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.664464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.664309 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-envoy\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.664464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.664389 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-data\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.664464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.664446 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-credential-socket\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.664699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.664677 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-certs\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.664766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.664703 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-socket\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.664766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.664710 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb25a1e-9f6b-4823-982a-82b25909fd53-istiod-ca-cert\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.666449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.666425 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-podinfo\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.666597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.666573 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-envoy\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.707005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.695079 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6phkn\" (UniqueName: \"kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-kube-api-access-6phkn\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.710070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.710028 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-token\") pod \"e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.737598 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.737566 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:26.857254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.857209 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:31:26.857446 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:26.857288 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:31:27.112419 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:27.112373 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt"] Oct 30 17:31:27.113346 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:31:27.113309 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb25a1e_9f6b_4823_982a_82b25909fd53.slice/crio-b3db9eb231aff2f892b036776e65c5d19c5790cc3d4e52b0e98c0b0ba9d4261c WatchSource:0}: Error finding container b3db9eb231aff2f892b036776e65c5d19c5790cc3d4e52b0e98c0b0ba9d4261c: Status 404 returned error can't find the container with id b3db9eb231aff2f892b036776e65c5d19c5790cc3d4e52b0e98c0b0ba9d4261c Oct 30 17:31:27.117018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:27.116981 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:31:27.117130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:27.117066 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:31:27.117130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:27.117114 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:31:27.584921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:27.580002 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:27.827923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:27.827377 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" event={"ID":"5eb25a1e-9f6b-4823-982a-82b25909fd53","Type":"ContainerStarted","Data":"060d6f1dd72bcd8266816e29075b16fb647825c7f84e68563a944b027dabee91"} Oct 30 17:31:27.827923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:27.827431 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" event={"ID":"5eb25a1e-9f6b-4823-982a-82b25909fd53","Type":"ContainerStarted","Data":"b3db9eb231aff2f892b036776e65c5d19c5790cc3d4e52b0e98c0b0ba9d4261c"} Oct 30 17:31:27.857612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:27.857471 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:31:27.857612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:27.857529 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:31:27.883936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:27.882129 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" podStartSLOduration=1.882107057 podStartE2EDuration="1.882107057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:27.874739212 +0000 UTC m=+2665.520249393" watchObservedRunningTime="2026-10-30 17:31:27.882107057 +0000 UTC m=+2665.527617237" Oct 30 17:31:28.131470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:28.131374 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:28.662819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:28.662775 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:28.752935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:28.750545 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:28.842934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:28.842453 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 17:31:28.842934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:28.842525 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" podUID="5eb25a1e-9f6b-4823-982a-82b25909fd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 17:31:28.968624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:28.942199 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:29.011771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:29.011728 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:31:29.018632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:29.018593 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:31:29.738436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:29.738403 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 17:31:29.738731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:29.738710 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" podUID="5eb25a1e-9f6b-4823-982a-82b25909fd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 17:31:29.856783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:29.856595 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:31:29.856783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:29.856683 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:31:29.956245 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:29.956199 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:30.741351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:30.741312 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 17:31:30.741642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:30.741614 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" podUID="5eb25a1e-9f6b-4823-982a-82b25909fd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 17:31:30.857410 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:30.857378 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 17:31:30.857841 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:30.857818 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 17:31:31.738818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:31.738779 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 17:31:31.739042 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:31.738835 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" podUID="5eb25a1e-9f6b-4823-982a-82b25909fd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 17:31:31.927505 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:31.927474 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:31:31.929383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:31.928479 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:31:31.934973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:31.934813 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:31:32.133979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:32.133944 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:32.426868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:32.426778 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:32.587516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:32.587475 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:32.738020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:32.737939 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 17:31:32.738020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:32.738002 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" podUID="5eb25a1e-9f6b-4823-982a-82b25909fd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 17:31:33.738578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:33.738539 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 30 17:31:33.739102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:33.738605 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" podUID="5eb25a1e-9f6b-4823-982a-82b25909fd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 30 17:31:34.278240 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:34.278200 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:34.441366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:34.441306 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:34.909528 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:34.909492 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf"] Oct 30 17:31:34.929220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:34.929186 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" Oct 30 17:31:34.942565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:34.942524 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf"] Oct 30 17:31:34.946753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:34.946718 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4fb821\"" Oct 30 17:31:34.965665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:34.965629 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4fb821-dockercfg-6x4xf\"" Oct 30 17:31:35.012403 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:35.012206 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:35.012849 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:35.012822 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:35.014230 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:35.014205 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:31:35.117072 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:35.116922 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9p5s\" (UniqueName: \"kubernetes.io/projected/785fe8ec-352e-4c68-ad24-924106f4dc53-kube-api-access-k9p5s\") pod \"payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf\" (UID: \"785fe8ec-352e-4c68-ad24-924106f4dc53\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" Oct 30 17:31:35.117773 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:35.117415 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785fe8ec-352e-4c68-ad24-924106f4dc53-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf\" (UID: \"785fe8ec-352e-4c68-ad24-924106f4dc53\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" Oct 30 17:31:35.219287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:35.219195 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9p5s\" (UniqueName: \"kubernetes.io/projected/785fe8ec-352e-4c68-ad24-924106f4dc53-kube-api-access-k9p5s\") pod \"payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf\" (UID: \"785fe8ec-352e-4c68-ad24-924106f4dc53\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" Oct 30 17:31:35.219475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:35.219387 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785fe8ec-352e-4c68-ad24-924106f4dc53-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf\" (UID: \"785fe8ec-352e-4c68-ad24-924106f4dc53\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" Oct 30 17:31:35.220267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:35.220239 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785fe8ec-352e-4c68-ad24-924106f4dc53-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf\" (UID: \"785fe8ec-352e-4c68-ad24-924106f4dc53\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" Oct 30 17:31:35.220915 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:35.220865 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:35.243966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:35.243938 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9p5s\" (UniqueName: \"kubernetes.io/projected/785fe8ec-352e-4c68-ad24-924106f4dc53-kube-api-access-k9p5s\") pod \"payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf\" (UID: \"785fe8ec-352e-4c68-ad24-924106f4dc53\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" Oct 30 17:31:35.263488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:35.263459 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" Oct 30 17:31:35.658701 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:31:35.657615 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785fe8ec_352e_4c68_ad24_924106f4dc53.slice/crio-00f1cbc40be038280d19270a117d092efe50c00c441f3adcba59566c48d63cb2 WatchSource:0}: Error finding container 00f1cbc40be038280d19270a117d092efe50c00c441f3adcba59566c48d63cb2: Status 404 returned error can't find the container with id 00f1cbc40be038280d19270a117d092efe50c00c441f3adcba59566c48d63cb2 Oct 30 17:31:35.663515 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:35.663486 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf"] Oct 30 17:31:36.405255 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:36.405210 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" event={"ID":"785fe8ec-352e-4c68-ad24-924106f4dc53","Type":"ContainerStarted","Data":"c1d1e11dbef03a81336c7b4426374eb0cc7b9d61ecd7df08712be60bafe34eaa"} Oct 30 17:31:36.405767 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:36.405278 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" event={"ID":"785fe8ec-352e-4c68-ad24-924106f4dc53","Type":"ContainerStarted","Data":"00f1cbc40be038280d19270a117d092efe50c00c441f3adcba59566c48d63cb2"} Oct 30 17:31:36.405767 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:36.405720 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" Oct 30 17:31:36.456147 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:36.456087 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" podStartSLOduration=2.456067064 podStartE2EDuration="2.456067064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:36.451635509 +0000 UTC m=+2674.097145689" watchObservedRunningTime="2026-10-30 17:31:36.456067064 +0000 UTC m=+2674.101577244" Oct 30 17:31:36.893747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:36.893699 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:37.912685 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:31:37.912649 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:31:38.508813 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:38.508761 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:38.680047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:38.679988 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:40.388920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:40.388354 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:42.489460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:42.489425 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:43.932010 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:43.931977 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:44.212790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:44.212491 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:44.316923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:44.297318 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4"] Oct 30 17:31:44.319259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:44.318965 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" podUID="d7a67ffe-2b3f-47f3-8c1a-386a57b32545" containerName="payload-pre-processing" containerID="cri-o://0ac50c50f6ecb1cd484c3a7d29f3ea94a3a6b707edf069cfe664619ef8b25c85" gracePeriod=30 Oct 30 17:31:44.375111 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:44.375057 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:44.742371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:44.742322 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:44.908197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:44.907743 4828 generic.go:356] "Generic (PLEG): container finished" podID="d7a67ffe-2b3f-47f3-8c1a-386a57b32545" containerID="0ac50c50f6ecb1cd484c3a7d29f3ea94a3a6b707edf069cfe664619ef8b25c85" exitCode=0 Oct 30 17:31:44.909659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:44.909622 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" event={"ID":"d7a67ffe-2b3f-47f3-8c1a-386a57b32545","Type":"ContainerDied","Data":"0ac50c50f6ecb1cd484c3a7d29f3ea94a3a6b707edf069cfe664619ef8b25c85"} Oct 30 17:31:45.031246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.031127 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:45.149302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.146119 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" Oct 30 17:31:45.192014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.191652 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:45.298999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.298965 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-kube-api-access-h6256\" (UniqueName: \"kubernetes.io/projected/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-kube-api-access-h6256\") pod \"d7a67ffe-2b3f-47f3-8c1a-386a57b32545\" (UID: \"d7a67ffe-2b3f-47f3-8c1a-386a57b32545\") " Oct 30 17:31:45.299217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.299064 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-plugins-config-volume\") pod \"d7a67ffe-2b3f-47f3-8c1a-386a57b32545\" (UID: \"d7a67ffe-2b3f-47f3-8c1a-386a57b32545\") " Oct 30 17:31:45.299440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.299410 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-plugins-config-volume" pod "d7a67ffe-2b3f-47f3-8c1a-386a57b32545" (UID: "d7a67ffe-2b3f-47f3-8c1a-386a57b32545"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:31:45.299731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.299709 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:31:45.301330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.301302 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-kube-api-access-h6256" pod "d7a67ffe-2b3f-47f3-8c1a-386a57b32545" (UID: "d7a67ffe-2b3f-47f3-8c1a-386a57b32545"). InnerVolumeSpecName "kube-api-access-h6256". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:45.401693 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.401652 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6256\" (UniqueName: \"kubernetes.io/projected/d7a67ffe-2b3f-47f3-8c1a-386a57b32545-kube-api-access-h6256\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:31:45.431705 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.431657 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:45.779527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.779456 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:45.919156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.919127 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" Oct 30 17:31:45.919337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.919163 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4" event={"ID":"d7a67ffe-2b3f-47f3-8c1a-386a57b32545","Type":"ContainerDied","Data":"d8ac4e7b094f75e8368975a61ab3151fb8e3f4a6a6b31220e501b03c03b59ed4"} Oct 30 17:31:45.919337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.919231 4828 scope.go:122] "RemoveContainer" containerID="0ac50c50f6ecb1cd484c3a7d29f3ea94a3a6b707edf069cfe664619ef8b25c85" Oct 30 17:31:45.937094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.937062 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:45.977493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.977455 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4"] Oct 30 17:31:45.983255 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:45.983225 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-abe243-854796fdf9-7khm4"] Oct 30 17:31:46.494104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:46.494068 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a67ffe-2b3f-47f3-8c1a-386a57b32545" path="/var/lib/kubelet/pods/d7a67ffe-2b3f-47f3-8c1a-386a57b32545/volumes" Oct 30 17:31:47.395514 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:47.395479 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:47.450151 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:47.450119 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" Oct 30 17:31:49.063275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:49.063236 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:50.700045 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:50.700009 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:50.851361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:50.851323 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:51.028237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:51.028201 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:51.942932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:51.939014 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:52.094888 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:52.094860 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:52.225168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:52.225078 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf"] Oct 30 17:31:52.225354 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:52.225331 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" containerID="cri-o://444800045b23640ee4cf6397fad6688befcbea4dd73f3755dbaf0e9da8b99782" gracePeriod=30 Oct 30 17:31:52.300803 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:52.300769 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:52.483977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:52.476445 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:54.673710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:54.673571 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:55.390336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:55.390298 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:56.390154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:56.390113 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:57.560199 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.560165 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:31:57.633617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.633584 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-envoy\") pod \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " Oct 30 17:31:57.633848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.633640 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istiod-ca-cert\") pod \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " Oct 30 17:31:57.633848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.633786 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-certs\") pod \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " Oct 30 17:31:57.633848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.633815 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-data\") pod \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " Oct 30 17:31:57.634129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.633866 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-token\" (UniqueName: \"kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-token\") pod \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " Oct 30 17:31:57.634129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.633923 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-kube-api-access-g7hxx\" (UniqueName: \"kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-kube-api-access-g7hxx\") pod \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " Oct 30 17:31:57.634129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.633959 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-podinfo\") pod \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " Oct 30 17:31:57.634129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.634041 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-credential-socket\") pod \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " Oct 30 17:31:57.634357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.634130 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-socket\") pod \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\" (UID: \"7e0b2768-4823-483d-91d6-b4bd5df2b78e\") " Oct 30 17:31:57.634357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.634132 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-certs" pod "7e0b2768-4823-483d-91d6-b4bd5df2b78e" (UID: "7e0b2768-4823-483d-91d6-b4bd5df2b78e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:57.634357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.634170 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-data" pod "7e0b2768-4823-483d-91d6-b4bd5df2b78e" (UID: "7e0b2768-4823-483d-91d6-b4bd5df2b78e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:57.634357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.634223 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istiod-ca-cert" pod "7e0b2768-4823-483d-91d6-b4bd5df2b78e" (UID: "7e0b2768-4823-483d-91d6-b4bd5df2b78e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:31:57.634581 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.634455 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-credential-socket" pod "7e0b2768-4823-483d-91d6-b4bd5df2b78e" (UID: "7e0b2768-4823-483d-91d6-b4bd5df2b78e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:57.634690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.634667 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-socket" pod "7e0b2768-4823-483d-91d6-b4bd5df2b78e" (UID: "7e0b2768-4823-483d-91d6-b4bd5df2b78e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:57.635071 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.635042 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:31:57.635176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.635074 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:31:57.635176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.635092 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:31:57.635176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.635106 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:31:57.635176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.635119 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:31:57.636340 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.636314 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-envoy" pod "7e0b2768-4823-483d-91d6-b4bd5df2b78e" (UID: "7e0b2768-4823-483d-91d6-b4bd5df2b78e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:31:57.636441 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.636364 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-kube-api-access-g7hxx" pod "7e0b2768-4823-483d-91d6-b4bd5df2b78e" (UID: "7e0b2768-4823-483d-91d6-b4bd5df2b78e"). InnerVolumeSpecName "kube-api-access-g7hxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:57.636581 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.636554 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-podinfo" pod "7e0b2768-4823-483d-91d6-b4bd5df2b78e" (UID: "7e0b2768-4823-483d-91d6-b4bd5df2b78e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:31:57.636672 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.636635 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-token" pod "7e0b2768-4823-483d-91d6-b4bd5df2b78e" (UID: "7e0b2768-4823-483d-91d6-b4bd5df2b78e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:31:57.737252 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.737211 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:31:57.737252 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.737241 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:31:57.737252 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.737252 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7hxx\" (UniqueName: \"kubernetes.io/projected/7e0b2768-4823-483d-91d6-b4bd5df2b78e-kube-api-access-g7hxx\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:31:57.737252 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:57.737263 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0b2768-4823-483d-91d6-b4bd5df2b78e-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:31:58.054086 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.054052 4828 generic.go:356] "Generic (PLEG): container finished" podID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerID="444800045b23640ee4cf6397fad6688befcbea4dd73f3755dbaf0e9da8b99782" exitCode=0 Oct 30 17:31:58.054272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.054102 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" Oct 30 17:31:58.054272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.054128 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" event={"ID":"7e0b2768-4823-483d-91d6-b4bd5df2b78e","Type":"ContainerDied","Data":"444800045b23640ee4cf6397fad6688befcbea4dd73f3755dbaf0e9da8b99782"} Oct 30 17:31:58.054272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.054212 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf" event={"ID":"7e0b2768-4823-483d-91d6-b4bd5df2b78e","Type":"ContainerDied","Data":"b486ed95716c8734749419fcd66c25076c1a0a50f336a7ef13dd82bea8541138"} Oct 30 17:31:58.054272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.054235 4828 scope.go:122] "RemoveContainer" containerID="444800045b23640ee4cf6397fad6688befcbea4dd73f3755dbaf0e9da8b99782" Oct 30 17:31:58.067978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.067953 4828 scope.go:122] "RemoveContainer" containerID="d7f5c6fdc49ca2b5a7b221cfa8a101002f6e42e9b840b073d7c7129932959155" Oct 30 17:31:58.079762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.079738 4828 scope.go:122] "RemoveContainer" containerID="444800045b23640ee4cf6397fad6688befcbea4dd73f3755dbaf0e9da8b99782" Oct 30 17:31:58.080208 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:31:58.080180 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444800045b23640ee4cf6397fad6688befcbea4dd73f3755dbaf0e9da8b99782\": container with ID starting with 444800045b23640ee4cf6397fad6688befcbea4dd73f3755dbaf0e9da8b99782 not found: ID does not exist" containerID="444800045b23640ee4cf6397fad6688befcbea4dd73f3755dbaf0e9da8b99782" Oct 30 17:31:58.080282 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.080223 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444800045b23640ee4cf6397fad6688befcbea4dd73f3755dbaf0e9da8b99782"} err="failed to get container status \"444800045b23640ee4cf6397fad6688befcbea4dd73f3755dbaf0e9da8b99782\": rpc error: code = NotFound desc = could not find container \"444800045b23640ee4cf6397fad6688befcbea4dd73f3755dbaf0e9da8b99782\": container with ID starting with 444800045b23640ee4cf6397fad6688befcbea4dd73f3755dbaf0e9da8b99782 not found: ID does not exist" Oct 30 17:31:58.080282 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.080252 4828 scope.go:122] "RemoveContainer" containerID="d7f5c6fdc49ca2b5a7b221cfa8a101002f6e42e9b840b073d7c7129932959155" Oct 30 17:31:58.080565 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:31:58.080545 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7f5c6fdc49ca2b5a7b221cfa8a101002f6e42e9b840b073d7c7129932959155\": container with ID starting with d7f5c6fdc49ca2b5a7b221cfa8a101002f6e42e9b840b073d7c7129932959155 not found: ID does not exist" containerID="d7f5c6fdc49ca2b5a7b221cfa8a101002f6e42e9b840b073d7c7129932959155" Oct 30 17:31:58.080678 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.080572 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7f5c6fdc49ca2b5a7b221cfa8a101002f6e42e9b840b073d7c7129932959155"} err="failed to get container status \"d7f5c6fdc49ca2b5a7b221cfa8a101002f6e42e9b840b073d7c7129932959155\": rpc error: code = NotFound desc = could not find container \"d7f5c6fdc49ca2b5a7b221cfa8a101002f6e42e9b840b073d7c7129932959155\": container with ID starting with d7f5c6fdc49ca2b5a7b221cfa8a101002f6e42e9b840b073d7c7129932959155 not found: ID does not exist" Oct 30 17:31:58.116709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.116673 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf"] Oct 30 17:31:58.126153 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.126120 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-abe243-openshift-default-5689d75869-m98bf"] Oct 30 17:31:58.219993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.210802 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:58.496210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:58.496120 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" path="/var/lib/kubelet/pods/7e0b2768-4823-483d-91d6-b4bd5df2b78e/volumes" Oct 30 17:31:59.624686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:59.624649 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:31:59.852391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:31:59.852225 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:00.231428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:00.231390 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:01.717534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:01.714554 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:03.703925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:03.697720 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:03.883215 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:03.883171 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:04.616470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:04.616425 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:05.097501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:05.097410 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:05.324620 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:05.324578 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:06.075623 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:06.075581 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:06.205817 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:06.205779 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc"] Oct 30 17:32:06.368056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:06.367970 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:07.296548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:07.296511 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:07.547190 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:07.547096 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:32:07.573985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:07.573956 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:32:08.234178 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:32:08.232673 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:32:08.582982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:08.582947 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:09.698517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:09.698478 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:09.866375 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:09.866333 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:10.780813 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:10.780775 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:11.206423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:11.206311 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" podUID="9df9a223-d981-4dc6-8dfa-4ba6985b08fe" containerName="payload-pre-processing" containerID="cri-o://f76f6df6170281b3ed463e99a7e0c07ef4f802135ab05527ed4b22977adc0343" gracePeriod=25 Oct 30 17:32:11.724589 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:11.713667 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:11.803941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:11.798163 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" Oct 30 17:32:11.829923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:11.820300 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-plugins-config-volume\") pod \"9df9a223-d981-4dc6-8dfa-4ba6985b08fe\" (UID: \"9df9a223-d981-4dc6-8dfa-4ba6985b08fe\") " Oct 30 17:32:11.829923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:11.820488 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-kube-api-access-49j4x\" (UniqueName: \"kubernetes.io/projected/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-kube-api-access-49j4x\") pod \"9df9a223-d981-4dc6-8dfa-4ba6985b08fe\" (UID: \"9df9a223-d981-4dc6-8dfa-4ba6985b08fe\") " Oct 30 17:32:11.878349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:11.876176 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-plugins-config-volume" pod "9df9a223-d981-4dc6-8dfa-4ba6985b08fe" (UID: "9df9a223-d981-4dc6-8dfa-4ba6985b08fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:32:11.934931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:11.934369 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:12.004925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:11.991706 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:12.232995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.215966 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk"] Oct 30 17:32:12.232995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.217394 4828 cpu_manager.go:414] "containerMap: removing container" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" Oct 30 17:32:12.232995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.217449 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" Oct 30 17:32:12.232995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.217488 4828 cpu_manager.go:414] "containerMap: removing container" podUID="9df9a223-d981-4dc6-8dfa-4ba6985b08fe" containerName="payload-pre-processing" Oct 30 17:32:12.232995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.217498 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df9a223-d981-4dc6-8dfa-4ba6985b08fe" containerName="payload-pre-processing" Oct 30 17:32:12.232995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.217586 4828 cpu_manager.go:414] "containerMap: removing container" podUID="d7a67ffe-2b3f-47f3-8c1a-386a57b32545" containerName="payload-pre-processing" Oct 30 17:32:12.232995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.217594 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a67ffe-2b3f-47f3-8c1a-386a57b32545" containerName="payload-pre-processing" Oct 30 17:32:12.232995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.217864 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" Oct 30 17:32:12.232995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.217890 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" Oct 30 17:32:12.232995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.217919 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df9a223-d981-4dc6-8dfa-4ba6985b08fe" containerName="payload-pre-processing" Oct 30 17:32:12.232995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.217940 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a67ffe-2b3f-47f3-8c1a-386a57b32545" containerName="payload-pre-processing" Oct 30 17:32:12.389470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.389431 4828 generic.go:356] "Generic (PLEG): container finished" podID="9df9a223-d981-4dc6-8dfa-4ba6985b08fe" containerID="f76f6df6170281b3ed463e99a7e0c07ef4f802135ab05527ed4b22977adc0343" exitCode=0 Oct 30 17:32:12.389816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.389796 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" Oct 30 17:32:12.395296 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.392369 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" event={"ID":"9df9a223-d981-4dc6-8dfa-4ba6985b08fe","Type":"ContainerDied","Data":"f76f6df6170281b3ed463e99a7e0c07ef4f802135ab05527ed4b22977adc0343"} Oct 30 17:32:12.395296 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.392424 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc" event={"ID":"9df9a223-d981-4dc6-8dfa-4ba6985b08fe","Type":"ContainerDied","Data":"a61492e5a460ec8d0e668039ddf4f995997a0493121b5f5c2358a3d9a7fc2d0d"} Oct 30 17:32:12.395296 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.392455 4828 scope.go:122] "RemoveContainer" containerID="f76f6df6170281b3ed463e99a7e0c07ef4f802135ab05527ed4b22977adc0343" Oct 30 17:32:12.395296 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.392657 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" Oct 30 17:32:12.395824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.395794 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-kube-api-access-49j4x" pod "9df9a223-d981-4dc6-8dfa-4ba6985b08fe" (UID: "9df9a223-d981-4dc6-8dfa-4ba6985b08fe"). InnerVolumeSpecName "kube-api-access-49j4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:12.408799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.408762 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk"] Oct 30 17:32:12.413485 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.413445 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a949e7-dockercfg-6bbsx\"" Oct 30 17:32:12.426649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.426604 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a949e7\"" Oct 30 17:32:12.428991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.428714 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nlxj\" (UniqueName: \"kubernetes.io/projected/727b046f-b952-46d6-ad72-c66eae2dac0b-kube-api-access-8nlxj\") pod \"payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk\" (UID: \"727b046f-b952-46d6-ad72-c66eae2dac0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" Oct 30 17:32:12.428991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.428834 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b046f-b952-46d6-ad72-c66eae2dac0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk\" (UID: \"727b046f-b952-46d6-ad72-c66eae2dac0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" Oct 30 17:32:12.429211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.429095 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49j4x\" (UniqueName: \"kubernetes.io/projected/9df9a223-d981-4dc6-8dfa-4ba6985b08fe-kube-api-access-49j4x\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:12.447800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.447771 4828 scope.go:122] "RemoveContainer" containerID="f76f6df6170281b3ed463e99a7e0c07ef4f802135ab05527ed4b22977adc0343" Oct 30 17:32:12.449215 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:32:12.449184 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76f6df6170281b3ed463e99a7e0c07ef4f802135ab05527ed4b22977adc0343\": container with ID starting with f76f6df6170281b3ed463e99a7e0c07ef4f802135ab05527ed4b22977adc0343 not found: ID does not exist" containerID="f76f6df6170281b3ed463e99a7e0c07ef4f802135ab05527ed4b22977adc0343" Oct 30 17:32:12.449332 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.449224 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76f6df6170281b3ed463e99a7e0c07ef4f802135ab05527ed4b22977adc0343"} err="failed to get container status \"f76f6df6170281b3ed463e99a7e0c07ef4f802135ab05527ed4b22977adc0343\": rpc error: code = NotFound desc = could not find container \"f76f6df6170281b3ed463e99a7e0c07ef4f802135ab05527ed4b22977adc0343\": container with ID starting with f76f6df6170281b3ed463e99a7e0c07ef4f802135ab05527ed4b22977adc0343 not found: ID does not exist" Oct 30 17:32:12.530832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.530793 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nlxj\" (UniqueName: \"kubernetes.io/projected/727b046f-b952-46d6-ad72-c66eae2dac0b-kube-api-access-8nlxj\") pod \"payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk\" (UID: \"727b046f-b952-46d6-ad72-c66eae2dac0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" Oct 30 17:32:12.531042 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.530982 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b046f-b952-46d6-ad72-c66eae2dac0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk\" (UID: \"727b046f-b952-46d6-ad72-c66eae2dac0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" Oct 30 17:32:12.531703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.531679 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b046f-b952-46d6-ad72-c66eae2dac0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk\" (UID: \"727b046f-b952-46d6-ad72-c66eae2dac0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" Oct 30 17:32:12.555430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.555399 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nlxj\" (UniqueName: \"kubernetes.io/projected/727b046f-b952-46d6-ad72-c66eae2dac0b-kube-api-access-8nlxj\") pod \"payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk\" (UID: \"727b046f-b952-46d6-ad72-c66eae2dac0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" Oct 30 17:32:12.739132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.739095 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc"] Oct 30 17:32:12.745830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.745800 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-5659c84bcf-c6mvc"] Oct 30 17:32:12.752443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:12.752417 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" Oct 30 17:32:13.070998 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:32:13.070957 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727b046f_b952_46d6_ad72_c66eae2dac0b.slice/crio-0262fad146d71a1a553bb8b12bbb610e6c820c527c400de54548cb2cc00677f1 WatchSource:0}: Error finding container 0262fad146d71a1a553bb8b12bbb610e6c820c527c400de54548cb2cc00677f1: Status 404 returned error can't find the container with id 0262fad146d71a1a553bb8b12bbb610e6c820c527c400de54548cb2cc00677f1 Oct 30 17:32:13.077561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:13.077536 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk"] Oct 30 17:32:13.405656 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:13.405548 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" event={"ID":"727b046f-b952-46d6-ad72-c66eae2dac0b","Type":"ContainerStarted","Data":"eadc6758d09874305a9c9f36968f6021d40ab258faea2c41fe05e4c4089db59b"} Oct 30 17:32:13.405866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:13.405657 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" Oct 30 17:32:13.405866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:13.405684 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" event={"ID":"727b046f-b952-46d6-ad72-c66eae2dac0b","Type":"ContainerStarted","Data":"0262fad146d71a1a553bb8b12bbb610e6c820c527c400de54548cb2cc00677f1"} Oct 30 17:32:13.449708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:13.449646 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" podStartSLOduration=2.449633502 podStartE2EDuration="2.449633502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:13.44823088 +0000 UTC m=+2711.093741073" watchObservedRunningTime="2026-10-30 17:32:13.449633502 +0000 UTC m=+2711.095143683" Oct 30 17:32:13.859650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:13.859600 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:14.492727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:14.492694 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df9a223-d981-4dc6-8dfa-4ba6985b08fe" path="/var/lib/kubelet/pods/9df9a223-d981-4dc6-8dfa-4ba6985b08fe/volumes" Oct 30 17:32:15.068220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:15.068167 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:15.259922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:15.248030 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:16.266937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:16.264711 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:17.970796 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:17.970756 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:19.222046 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.221988 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk"] Oct 30 17:32:19.223436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.222371 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" podUID="af3275f8-272d-40b0-84e1-d3033d1697e4" containerName="maas-api" containerID="cri-o://421bf53d4ed112be66d36b3bebb311f8f7632b19eddc42b0424fc21bf16c35af" gracePeriod=30 Oct 30 17:32:19.314433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.314394 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:19.492374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.492294 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:19.577455 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.577089 4828 generic.go:356] "Generic (PLEG): container finished" podID="af3275f8-272d-40b0-84e1-d3033d1697e4" containerID="421bf53d4ed112be66d36b3bebb311f8f7632b19eddc42b0424fc21bf16c35af" exitCode=0 Oct 30 17:32:19.577455 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.577384 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" event={"ID":"af3275f8-272d-40b0-84e1-d3033d1697e4","Type":"ContainerDied","Data":"421bf53d4ed112be66d36b3bebb311f8f7632b19eddc42b0424fc21bf16c35af"} Oct 30 17:32:19.643793 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.643759 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:19.653706 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.653678 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:32:19.665435 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.665397 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3275f8-272d-40b0-84e1-d3033d1697e4-kube-api-access-pbbp7\" (UniqueName: \"kubernetes.io/projected/af3275f8-272d-40b0-84e1-d3033d1697e4-kube-api-access-pbbp7\") pod \"af3275f8-272d-40b0-84e1-d3033d1697e4\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " Oct 30 17:32:19.665640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.665464 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-maas-api-tls\") pod \"af3275f8-272d-40b0-84e1-d3033d1697e4\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " Oct 30 17:32:19.665705 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.665674 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-metrics-certs\") pod \"af3275f8-272d-40b0-84e1-d3033d1697e4\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " Oct 30 17:32:19.665776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.665730 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af3275f8-272d-40b0-84e1-d3033d1697e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3275f8-272d-40b0-84e1-d3033d1697e4-openshift-service-ca\") pod \"af3275f8-272d-40b0-84e1-d3033d1697e4\" (UID: \"af3275f8-272d-40b0-84e1-d3033d1697e4\") " Oct 30 17:32:19.684022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.670581 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3275f8-272d-40b0-84e1-d3033d1697e4-openshift-service-ca" pod "af3275f8-272d-40b0-84e1-d3033d1697e4" (UID: "af3275f8-272d-40b0-84e1-d3033d1697e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:32:19.696922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.691153 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-maas-api-tls" pod "af3275f8-272d-40b0-84e1-d3033d1697e4" (UID: "af3275f8-272d-40b0-84e1-d3033d1697e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:32:19.837533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.826170 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3275f8-272d-40b0-84e1-d3033d1697e4-kube-api-access-pbbp7" pod "af3275f8-272d-40b0-84e1-d3033d1697e4" (UID: "af3275f8-272d-40b0-84e1-d3033d1697e4"). InnerVolumeSpecName "kube-api-access-pbbp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:19.862825 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.862022 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-metrics-certs" pod "af3275f8-272d-40b0-84e1-d3033d1697e4" (UID: "af3275f8-272d-40b0-84e1-d3033d1697e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:32:19.863099 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.830327 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af3275f8-272d-40b0-84e1-d3033d1697e4-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:19.997918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.997857 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbbp7\" (UniqueName: \"kubernetes.io/projected/af3275f8-272d-40b0-84e1-d3033d1697e4-kube-api-access-pbbp7\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:20.003292 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:19.999657 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:20.004382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:20.004359 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af3275f8-272d-40b0-84e1-d3033d1697e4-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:20.161691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:20.161600 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:20.337924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:20.337263 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:20.484361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:20.484257 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:20.588941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:20.588872 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" Oct 30 17:32:20.589149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:20.588939 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk" event={"ID":"af3275f8-272d-40b0-84e1-d3033d1697e4","Type":"ContainerDied","Data":"865aa9aa493d4451e53c6d229e51d672c114d8fe646b554bdbb7fca5b3a1d5e1"} Oct 30 17:32:20.589149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:20.589003 4828 scope.go:122] "RemoveContainer" containerID="421bf53d4ed112be66d36b3bebb311f8f7632b19eddc42b0424fc21bf16c35af" Oct 30 17:32:20.645826 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:20.645795 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk"] Oct 30 17:32:20.658532 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:20.658502 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-63e60d-584464454b-gntmk"] Oct 30 17:32:20.792955 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:20.792923 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:22.166126 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:22.166068 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:22.495649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:22.495562 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af3275f8-272d-40b0-84e1-d3033d1697e4" path="/var/lib/kubelet/pods/af3275f8-272d-40b0-84e1-d3033d1697e4/volumes" Oct 30 17:32:23.950483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:23.947148 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:24.421248 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:24.421215 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" Oct 30 17:32:25.505524 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:25.505464 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:27.274928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:27.274870 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:27.333628 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:27.332601 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6"] Oct 30 17:32:27.333628 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:27.332875 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" podUID="44a0eb80-9692-4e67-a75b-43f638a8177c" containerName="istio-proxy" containerID="cri-o://63ec38b2f9d0b6ff3d59c68c40f1739487206e70817597bf63a8d60949b8c8e8" gracePeriod=30 Oct 30 17:32:27.347237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:27.346134 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq"] Oct 30 17:32:27.347237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:27.346404 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" containerID="cri-o://c61a207b0930f0a675be70d947eeaa1d874239d4cae15cd9e4fac5a961c376c4" gracePeriod=30 Oct 30 17:32:27.362066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:27.362016 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7"] Oct 30 17:32:27.369035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:27.364405 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" podUID="25103151-7048-4796-adab-7e5b0b971e7a" containerName="istio-proxy" containerID="cri-o://1a3c0bafc5b2e01c43d4a68e21febaa04248ff235243c483bd4b045fae241a24" gracePeriod=30 Oct 30 17:32:27.855730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:27.855696 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:28.020608 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:28.020568 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:28.201433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:28.201340 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:28.497980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:28.493514 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:29.200880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:29.200838 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:30.778152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.778118 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4"] Oct 30 17:32:30.779264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.779239 4828 cpu_manager.go:414] "containerMap: removing container" podUID="af3275f8-272d-40b0-84e1-d3033d1697e4" containerName="maas-api" Oct 30 17:32:30.779264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.779263 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3275f8-272d-40b0-84e1-d3033d1697e4" containerName="maas-api" Oct 30 17:32:30.779446 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.779298 4828 cpu_manager.go:414] "containerMap: removing container" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" Oct 30 17:32:30.779446 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.779305 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0b2768-4823-483d-91d6-b4bd5df2b78e" containerName="istio-proxy" Oct 30 17:32:30.779557 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.779540 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3275f8-272d-40b0-84e1-d3033d1697e4" containerName="maas-api" Oct 30 17:32:30.784839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.784816 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:30.805141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.805110 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4"] Oct 30 17:32:30.807145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.807122 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4fb821-serving-cert\"" Oct 30 17:32:30.909150 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.909112 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:30.967863 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.967825 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:30.968074 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.967880 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcmgg\" (UniqueName: \"kubernetes.io/projected/83781a17-2754-43ad-9832-6bf1700b0eeb-kube-api-access-pcmgg\") pod \"maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:30.968194 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.968172 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-metrics-certs\") pod \"maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:30.968272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:30.968259 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83781a17-2754-43ad-9832-6bf1700b0eeb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:31.073275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:31.073175 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcmgg\" (UniqueName: \"kubernetes.io/projected/83781a17-2754-43ad-9832-6bf1700b0eeb-kube-api-access-pcmgg\") pod \"maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:31.073483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:31.073459 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-metrics-certs\") pod \"maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:31.073563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:31.073530 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83781a17-2754-43ad-9832-6bf1700b0eeb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:31.073799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:31.073769 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:31.074772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:31.074746 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83781a17-2754-43ad-9832-6bf1700b0eeb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:31.093829 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:31.093635 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:31.099292 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:31.099251 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-metrics-certs\") pod \"maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:31.105397 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:31.105358 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:31.129501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:31.129465 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcmgg\" (UniqueName: \"kubernetes.io/projected/83781a17-2754-43ad-9832-6bf1700b0eeb-kube-api-access-pcmgg\") pod \"maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:31.401246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:31.401159 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:31.721027 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:32:31.719050 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83781a17_2754_43ad_9832_6bf1700b0eeb.slice/crio-5236ee8b8c4ac733a9156d6e55a364937a9eadea0e8eddbd48c5246f8ef971c8 WatchSource:0}: Error finding container 5236ee8b8c4ac733a9156d6e55a364937a9eadea0e8eddbd48c5246f8ef971c8: Status 404 returned error can't find the container with id 5236ee8b8c4ac733a9156d6e55a364937a9eadea0e8eddbd48c5246f8ef971c8 Oct 30 17:32:31.721027 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:31.720634 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4"] Oct 30 17:32:31.721027 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:31.720691 4828 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:32:32.626422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.626354 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:32:32.726484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.726449 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/383c4614-f032-4952-9be1-d69075b86db8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383c4614-f032-4952-9be1-d69075b86db8-istio-podinfo\") pod \"383c4614-f032-4952-9be1-d69075b86db8\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " Oct 30 17:32:32.728169 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.728140 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-data\") pod \"383c4614-f032-4952-9be1-d69075b86db8\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " Oct 30 17:32:32.728426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.728407 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-credential-socket\") pod \"383c4614-f032-4952-9be1-d69075b86db8\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " Oct 30 17:32:32.728565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.728550 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/383c4614-f032-4952-9be1-d69075b86db8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383c4614-f032-4952-9be1-d69075b86db8-istiod-ca-cert\") pod \"383c4614-f032-4952-9be1-d69075b86db8\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " Oct 30 17:32:32.728732 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.728719 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-socket\") pod \"383c4614-f032-4952-9be1-d69075b86db8\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " Oct 30 17:32:32.728836 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.728823 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-certs\") pod \"383c4614-f032-4952-9be1-d69075b86db8\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " Oct 30 17:32:32.728998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.728983 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-istio-token\" (UniqueName: \"kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-istio-token\") pod \"383c4614-f032-4952-9be1-d69075b86db8\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " Oct 30 17:32:32.733117 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.730080 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-socket" pod "383c4614-f032-4952-9be1-d69075b86db8" (UID: "383c4614-f032-4952-9be1-d69075b86db8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.733242 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.730241 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383c4614-f032-4952-9be1-d69075b86db8-istiod-ca-cert" pod "383c4614-f032-4952-9be1-d69075b86db8" (UID: "383c4614-f032-4952-9be1-d69075b86db8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:32:32.733242 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.730472 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-credential-socket" pod "383c4614-f032-4952-9be1-d69075b86db8" (UID: "383c4614-f032-4952-9be1-d69075b86db8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.733242 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.731410 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-certs" pod "383c4614-f032-4952-9be1-d69075b86db8" (UID: "383c4614-f032-4952-9be1-d69075b86db8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.733242 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.731912 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-data" pod "383c4614-f032-4952-9be1-d69075b86db8" (UID: "383c4614-f032-4952-9be1-d69075b86db8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.734240 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.734220 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-envoy\") pod \"383c4614-f032-4952-9be1-d69075b86db8\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " Oct 30 17:32:32.743201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.743177 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-kube-api-access-f5jcp\" (UniqueName: \"kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-kube-api-access-f5jcp\") pod \"383c4614-f032-4952-9be1-d69075b86db8\" (UID: \"383c4614-f032-4952-9be1-d69075b86db8\") " Oct 30 17:32:32.743882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.743757 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-istio-token" pod "383c4614-f032-4952-9be1-d69075b86db8" (UID: "383c4614-f032-4952-9be1-d69075b86db8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:32.744477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.744435 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/383c4614-f032-4952-9be1-d69075b86db8-istio-podinfo" pod "383c4614-f032-4952-9be1-d69075b86db8" (UID: "383c4614-f032-4952-9be1-d69075b86db8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:32:32.744675 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.744653 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-envoy" pod "383c4614-f032-4952-9be1-d69075b86db8" (UID: "383c4614-f032-4952-9be1-d69075b86db8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.747336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.747311 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383c4614-f032-4952-9be1-d69075b86db8-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:32.747484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.747470 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:32.747616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.747603 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:32.747741 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.747712 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383c4614-f032-4952-9be1-d69075b86db8-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:32.747851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.747839 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:32.748078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.748063 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:32.748212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.748198 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:32.748317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.748305 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383c4614-f032-4952-9be1-d69075b86db8-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:32.748444 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.747789 4828 generic.go:356] "Generic (PLEG): container finished" podID="44a0eb80-9692-4e67-a75b-43f638a8177c" containerID="63ec38b2f9d0b6ff3d59c68c40f1739487206e70817597bf63a8d60949b8c8e8" exitCode=0 Oct 30 17:32:32.748692 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.747812 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" event={"ID":"44a0eb80-9692-4e67-a75b-43f638a8177c","Type":"ContainerDied","Data":"63ec38b2f9d0b6ff3d59c68c40f1739487206e70817597bf63a8d60949b8c8e8"} Oct 30 17:32:32.748871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.747373 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-kube-api-access-f5jcp" pod "383c4614-f032-4952-9be1-d69075b86db8" (UID: "383c4614-f032-4952-9be1-d69075b86db8"). InnerVolumeSpecName "kube-api-access-f5jcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:32.754932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.751407 4828 generic.go:356] "Generic (PLEG): container finished" podID="383c4614-f032-4952-9be1-d69075b86db8" containerID="c61a207b0930f0a675be70d947eeaa1d874239d4cae15cd9e4fac5a961c376c4" exitCode=0 Oct 30 17:32:32.754932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.751494 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" event={"ID":"383c4614-f032-4952-9be1-d69075b86db8","Type":"ContainerDied","Data":"c61a207b0930f0a675be70d947eeaa1d874239d4cae15cd9e4fac5a961c376c4"} Oct 30 17:32:32.754932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.751542 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" event={"ID":"383c4614-f032-4952-9be1-d69075b86db8","Type":"ContainerDied","Data":"e0d369f3c4d3fd96ae0ecb4e9ab6ce96d696b91fc18f5ae012a57def7dded41d"} Oct 30 17:32:32.754932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.751571 4828 scope.go:122] "RemoveContainer" containerID="c61a207b0930f0a675be70d947eeaa1d874239d4cae15cd9e4fac5a961c376c4" Oct 30 17:32:32.754932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.751740 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq" Oct 30 17:32:32.773549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.773516 4828 generic.go:356] "Generic (PLEG): container finished" podID="25103151-7048-4796-adab-7e5b0b971e7a" containerID="1a3c0bafc5b2e01c43d4a68e21febaa04248ff235243c483bd4b045fae241a24" exitCode=0 Oct 30 17:32:32.773770 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.773748 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" event={"ID":"25103151-7048-4796-adab-7e5b0b971e7a","Type":"ContainerDied","Data":"1a3c0bafc5b2e01c43d4a68e21febaa04248ff235243c483bd4b045fae241a24"} Oct 30 17:32:32.776128 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.776097 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" event={"ID":"83781a17-2754-43ad-9832-6bf1700b0eeb","Type":"ContainerStarted","Data":"ab155f5c3760503f447353a4844f9009e55727ee4d9b8aa6a3cc7e93fa979947"} Oct 30 17:32:32.776213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.776162 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" event={"ID":"83781a17-2754-43ad-9832-6bf1700b0eeb","Type":"ContainerStarted","Data":"5236ee8b8c4ac733a9156d6e55a364937a9eadea0e8eddbd48c5246f8ef971c8"} Oct 30 17:32:32.777358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.777332 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:32.789341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.789317 4828 scope.go:122] "RemoveContainer" containerID="81e498bc4f22ef97f4e982dec3ad06d920c8ac1f8f8728ab29e2e5cf6a701c8c" Oct 30 17:32:32.830674 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.830613 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" podStartSLOduration=2.423887381 podStartE2EDuration="2.830593407s" totalImagesPullingTime="406.706026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:32:32.822035621 +0000 UTC m=+2730.467545800" watchObservedRunningTime="2026-10-30 17:32:32.830593407 +0000 UTC m=+2730.476103589" Oct 30 17:32:32.835118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.834997 4828 scope.go:122] "RemoveContainer" containerID="c61a207b0930f0a675be70d947eeaa1d874239d4cae15cd9e4fac5a961c376c4" Oct 30 17:32:32.835378 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:32:32.835351 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61a207b0930f0a675be70d947eeaa1d874239d4cae15cd9e4fac5a961c376c4\": container with ID starting with c61a207b0930f0a675be70d947eeaa1d874239d4cae15cd9e4fac5a961c376c4 not found: ID does not exist" containerID="c61a207b0930f0a675be70d947eeaa1d874239d4cae15cd9e4fac5a961c376c4" Oct 30 17:32:32.835534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.835506 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61a207b0930f0a675be70d947eeaa1d874239d4cae15cd9e4fac5a961c376c4"} err="failed to get container status \"c61a207b0930f0a675be70d947eeaa1d874239d4cae15cd9e4fac5a961c376c4\": rpc error: code = NotFound desc = could not find container \"c61a207b0930f0a675be70d947eeaa1d874239d4cae15cd9e4fac5a961c376c4\": container with ID starting with c61a207b0930f0a675be70d947eeaa1d874239d4cae15cd9e4fac5a961c376c4 not found: ID does not exist" Oct 30 17:32:32.835642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.835630 4828 scope.go:122] "RemoveContainer" containerID="81e498bc4f22ef97f4e982dec3ad06d920c8ac1f8f8728ab29e2e5cf6a701c8c" Oct 30 17:32:32.838845 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:32:32.837388 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e498bc4f22ef97f4e982dec3ad06d920c8ac1f8f8728ab29e2e5cf6a701c8c\": container with ID starting with 81e498bc4f22ef97f4e982dec3ad06d920c8ac1f8f8728ab29e2e5cf6a701c8c not found: ID does not exist" containerID="81e498bc4f22ef97f4e982dec3ad06d920c8ac1f8f8728ab29e2e5cf6a701c8c" Oct 30 17:32:32.838845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.837430 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e498bc4f22ef97f4e982dec3ad06d920c8ac1f8f8728ab29e2e5cf6a701c8c"} err="failed to get container status \"81e498bc4f22ef97f4e982dec3ad06d920c8ac1f8f8728ab29e2e5cf6a701c8c\": rpc error: code = NotFound desc = could not find container \"81e498bc4f22ef97f4e982dec3ad06d920c8ac1f8f8728ab29e2e5cf6a701c8c\": container with ID starting with 81e498bc4f22ef97f4e982dec3ad06d920c8ac1f8f8728ab29e2e5cf6a701c8c not found: ID does not exist" Oct 30 17:32:32.843481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.843456 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:32:32.846472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.846453 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:32:32.850816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.850788 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5jcp\" (UniqueName: \"kubernetes.io/projected/383c4614-f032-4952-9be1-d69075b86db8-kube-api-access-f5jcp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:32.872193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.872155 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq"] Oct 30 17:32:32.880064 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.880032 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-jv5dq"] Oct 30 17:32:32.952127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952037 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-socket\") pod \"44a0eb80-9692-4e67-a75b-43f638a8177c\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " Oct 30 17:32:32.952127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952087 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-istio-token\" (UniqueName: \"kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-istio-token\") pod \"25103151-7048-4796-adab-7e5b0b971e7a\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " Oct 30 17:32:32.952127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952119 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-envoy\") pod \"44a0eb80-9692-4e67-a75b-43f638a8177c\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " Oct 30 17:32:32.952428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952168 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-certs\") pod \"44a0eb80-9692-4e67-a75b-43f638a8177c\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " Oct 30 17:32:32.952428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952188 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-kube-api-access-ctz6k\" (UniqueName: \"kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-kube-api-access-ctz6k\") pod \"25103151-7048-4796-adab-7e5b0b971e7a\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " Oct 30 17:32:32.952428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952327 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-data\") pod \"25103151-7048-4796-adab-7e5b0b971e7a\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " Oct 30 17:32:32.952428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952361 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-kube-api-access-nvs52\" (UniqueName: \"kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-kube-api-access-nvs52\") pod \"44a0eb80-9692-4e67-a75b-43f638a8177c\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " Oct 30 17:32:32.952428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952391 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25103151-7048-4796-adab-7e5b0b971e7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25103151-7048-4796-adab-7e5b0b971e7a-istiod-ca-cert\") pod \"25103151-7048-4796-adab-7e5b0b971e7a\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " Oct 30 17:32:32.952768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952434 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25103151-7048-4796-adab-7e5b0b971e7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25103151-7048-4796-adab-7e5b0b971e7a-istio-podinfo\") pod \"25103151-7048-4796-adab-7e5b0b971e7a\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " Oct 30 17:32:32.952768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952469 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-envoy\") pod \"25103151-7048-4796-adab-7e5b0b971e7a\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " Oct 30 17:32:32.952768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952498 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-certs\") pod \"25103151-7048-4796-adab-7e5b0b971e7a\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " Oct 30 17:32:32.952768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952527 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-socket\") pod \"25103151-7048-4796-adab-7e5b0b971e7a\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " Oct 30 17:32:32.952768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952576 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-token\" (UniqueName: \"kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-token\") pod \"44a0eb80-9692-4e67-a75b-43f638a8177c\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " Oct 30 17:32:32.952768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952601 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-credential-socket\") pod \"44a0eb80-9692-4e67-a75b-43f638a8177c\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " Oct 30 17:32:32.952768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952655 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a0eb80-9692-4e67-a75b-43f638a8177c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a0eb80-9692-4e67-a75b-43f638a8177c-istiod-ca-cert\") pod \"44a0eb80-9692-4e67-a75b-43f638a8177c\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " Oct 30 17:32:32.952768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952697 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-credential-socket\") pod \"25103151-7048-4796-adab-7e5b0b971e7a\" (UID: \"25103151-7048-4796-adab-7e5b0b971e7a\") " Oct 30 17:32:32.952768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952745 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-data\") pod \"44a0eb80-9692-4e67-a75b-43f638a8177c\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " Oct 30 17:32:32.953322 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.952778 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-podinfo\") pod \"44a0eb80-9692-4e67-a75b-43f638a8177c\" (UID: \"44a0eb80-9692-4e67-a75b-43f638a8177c\") " Oct 30 17:32:32.953962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.953885 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-certs" pod "44a0eb80-9692-4e67-a75b-43f638a8177c" (UID: "44a0eb80-9692-4e67-a75b-43f638a8177c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.954361 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.954338 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-socket" pod "44a0eb80-9692-4e67-a75b-43f638a8177c" (UID: "44a0eb80-9692-4e67-a75b-43f638a8177c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.954503 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.954478 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-certs" pod "25103151-7048-4796-adab-7e5b0b971e7a" (UID: "25103151-7048-4796-adab-7e5b0b971e7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.954834 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.954807 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a0eb80-9692-4e67-a75b-43f638a8177c-istiod-ca-cert" pod "44a0eb80-9692-4e67-a75b-43f638a8177c" (UID: "44a0eb80-9692-4e67-a75b-43f638a8177c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:32:32.955304 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.955125 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-credential-socket" pod "25103151-7048-4796-adab-7e5b0b971e7a" (UID: "25103151-7048-4796-adab-7e5b0b971e7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.955674 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.955448 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-credential-socket" pod "44a0eb80-9692-4e67-a75b-43f638a8177c" (UID: "44a0eb80-9692-4e67-a75b-43f638a8177c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.955950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.955875 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-data" pod "44a0eb80-9692-4e67-a75b-43f638a8177c" (UID: "44a0eb80-9692-4e67-a75b-43f638a8177c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.956060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.956022 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-podinfo" pod "44a0eb80-9692-4e67-a75b-43f638a8177c" (UID: "44a0eb80-9692-4e67-a75b-43f638a8177c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:32:32.956634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.956593 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25103151-7048-4796-adab-7e5b0b971e7a-istio-podinfo" pod "25103151-7048-4796-adab-7e5b0b971e7a" (UID: "25103151-7048-4796-adab-7e5b0b971e7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:32:32.957478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.957087 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-envoy" pod "25103151-7048-4796-adab-7e5b0b971e7a" (UID: "25103151-7048-4796-adab-7e5b0b971e7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.957478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.957281 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-socket" pod "25103151-7048-4796-adab-7e5b0b971e7a" (UID: "25103151-7048-4796-adab-7e5b0b971e7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.957478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.957441 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25103151-7048-4796-adab-7e5b0b971e7a-istiod-ca-cert" pod "25103151-7048-4796-adab-7e5b0b971e7a" (UID: "25103151-7048-4796-adab-7e5b0b971e7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:32:32.957478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.957446 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-data" pod "25103151-7048-4796-adab-7e5b0b971e7a" (UID: "25103151-7048-4796-adab-7e5b0b971e7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:32.957821 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.957684 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-istio-token" pod "25103151-7048-4796-adab-7e5b0b971e7a" (UID: "25103151-7048-4796-adab-7e5b0b971e7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:32.958264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.958231 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-token" pod "44a0eb80-9692-4e67-a75b-43f638a8177c" (UID: "44a0eb80-9692-4e67-a75b-43f638a8177c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:32.958366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.958334 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-kube-api-access-ctz6k" pod "25103151-7048-4796-adab-7e5b0b971e7a" (UID: "25103151-7048-4796-adab-7e5b0b971e7a"). InnerVolumeSpecName "kube-api-access-ctz6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:32.959175 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.959135 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-kube-api-access-nvs52" pod "44a0eb80-9692-4e67-a75b-43f638a8177c" (UID: "44a0eb80-9692-4e67-a75b-43f638a8177c"). InnerVolumeSpecName "kube-api-access-nvs52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:32:32.960617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:32.960592 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-envoy" pod "44a0eb80-9692-4e67-a75b-43f638a8177c" (UID: "44a0eb80-9692-4e67-a75b-43f638a8177c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:32:33.013261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.013159 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:33.054689 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054651 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054689 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054689 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054699 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054708 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054717 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a0eb80-9692-4e67-a75b-43f638a8177c-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054727 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054735 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054745 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054752 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054761 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054773 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054781 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a0eb80-9692-4e67-a75b-43f638a8177c-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054789 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctz6k\" (UniqueName: \"kubernetes.io/projected/25103151-7048-4796-adab-7e5b0b971e7a-kube-api-access-ctz6k\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054798 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054806 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvs52\" (UniqueName: \"kubernetes.io/projected/44a0eb80-9692-4e67-a75b-43f638a8177c-kube-api-access-nvs52\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054814 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25103151-7048-4796-adab-7e5b0b971e7a-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054823 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25103151-7048-4796-adab-7e5b0b971e7a-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.054914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.054832 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25103151-7048-4796-adab-7e5b0b971e7a-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:32:33.496713 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.496682 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mj8j7_c055e279-12c7-4707-b602-56548f5df870/payload-processing/0.log" Oct 30 17:32:33.705388 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.705353 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mj8j7_c055e279-12c7-4707-b602-56548f5df870/payload-processing/0.log" Oct 30 17:32:33.785724 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.785655 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" Oct 30 17:32:33.785724 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.785693 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6" event={"ID":"44a0eb80-9692-4e67-a75b-43f638a8177c","Type":"ContainerDied","Data":"9d23f01431dad247dbf0396b5269a8f0ded1390ad470a33c6413c1e3c155071a"} Oct 30 17:32:33.786078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.785778 4828 scope.go:122] "RemoveContainer" containerID="63ec38b2f9d0b6ff3d59c68c40f1739487206e70817597bf63a8d60949b8c8e8" Oct 30 17:32:33.793947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.788861 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" Oct 30 17:32:33.793947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.788941 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7" event={"ID":"25103151-7048-4796-adab-7e5b0b971e7a","Type":"ContainerDied","Data":"c57152bbc73767eb13aa1eb04f6b549ddb9ea759763373c3636502b01df7b84d"} Oct 30 17:32:33.800024 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.800002 4828 scope.go:122] "RemoveContainer" containerID="1a3c0bafc5b2e01c43d4a68e21febaa04248ff235243c483bd4b045fae241a24" Oct 30 17:32:33.857366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.857329 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6"] Oct 30 17:32:33.869683 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.869646 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-kncm6"] Oct 30 17:32:33.898053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.898013 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7"] Oct 30 17:32:33.902726 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:33.902693 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-76nb7"] Oct 30 17:32:34.495073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:34.495036 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25103151-7048-4796-adab-7e5b0b971e7a" path="/var/lib/kubelet/pods/25103151-7048-4796-adab-7e5b0b971e7a/volumes" Oct 30 17:32:34.495587 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:34.495569 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="383c4614-f032-4952-9be1-d69075b86db8" path="/var/lib/kubelet/pods/383c4614-f032-4952-9be1-d69075b86db8/volumes" Oct 30 17:32:34.496156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:34.496139 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a0eb80-9692-4e67-a75b-43f638a8177c" path="/var/lib/kubelet/pods/44a0eb80-9692-4e67-a75b-43f638a8177c/volumes" Oct 30 17:32:35.259933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:35.259876 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:37.447184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:37.447141 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:37.913523 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:32:37.913486 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:32:38.842519 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:38.842468 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw"] Oct 30 17:32:39.033559 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:39.033520 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:39.207832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:39.207737 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:39.813655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:39.813622 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:32:40.929105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:40.926366 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:43.159943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:43.159886 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:43.843616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:43.843572 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" podUID="68e707eb-e7e5-4464-863d-89f0e71fe2a5" containerName="payload-processing" containerID="cri-o://46b68ab8b5e66d80341a9a9c6f63ef94cc8e0f538f6025bc5ba1ba0b880bdb6e" gracePeriod=25 Oct 30 17:32:44.854564 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:44.854522 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:46.993925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:46.991560 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:48.723145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:48.721546 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:49.022882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:49.022844 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" podUID="68e707eb-e7e5-4464-863d-89f0e71fe2a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:32:50.351602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:50.351565 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:52.224933 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:52.224874 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:54.255733 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:54.255681 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:55.082779 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:55.082736 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:55.234589 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:55.234541 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:55.433110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:55.433015 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:55.985166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:55.982167 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:56.689291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:56.689259 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:56.903129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:56.903096 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:57.346775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:57.346742 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:59.023626 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:59.023590 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" podUID="68e707eb-e7e5-4464-863d-89f0e71fe2a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:32:59.311157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:59.311075 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:32:59.757479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:32:59.757387 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:00.357421 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:00.357380 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:00.581480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:00.581432 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:00.821963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:00.821678 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:01.067069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.055961 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4"] Oct 30 17:33:01.070920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.068064 4828 cpu_manager.go:414] "containerMap: removing container" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" Oct 30 17:33:01.070920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.068090 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" Oct 30 17:33:01.070920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.068120 4828 cpu_manager.go:414] "containerMap: removing container" podUID="44a0eb80-9692-4e67-a75b-43f638a8177c" containerName="istio-proxy" Oct 30 17:33:01.070920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.068127 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a0eb80-9692-4e67-a75b-43f638a8177c" containerName="istio-proxy" Oct 30 17:33:01.070920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.068145 4828 cpu_manager.go:414] "containerMap: removing container" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" Oct 30 17:33:01.070920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.068152 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" Oct 30 17:33:01.070920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.068168 4828 cpu_manager.go:414] "containerMap: removing container" podUID="25103151-7048-4796-adab-7e5b0b971e7a" containerName="istio-proxy" Oct 30 17:33:01.070920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.068174 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25103151-7048-4796-adab-7e5b0b971e7a" containerName="istio-proxy" Oct 30 17:33:01.070920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.068497 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" Oct 30 17:33:01.070920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.068517 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a0eb80-9692-4e67-a75b-43f638a8177c" containerName="istio-proxy" Oct 30 17:33:01.070920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.068528 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="383c4614-f032-4952-9be1-d69075b86db8" containerName="istio-proxy" Oct 30 17:33:01.070920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.068540 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="25103151-7048-4796-adab-7e5b0b971e7a" containerName="istio-proxy" Oct 30 17:33:01.149097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.107476 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" Oct 30 17:33:01.149097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.109228 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4"] Oct 30 17:33:01.164934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.161848 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3260f8\"" Oct 30 17:33:01.164934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.162786 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3260f8-dockercfg-n8z9l\"" Oct 30 17:33:01.199192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.199154 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzntf\" (UniqueName: \"kubernetes.io/projected/6b374e83-1270-47c2-9620-88d9c947646d-kube-api-access-zzntf\") pod \"payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4\" (UID: \"6b374e83-1270-47c2-9620-88d9c947646d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" Oct 30 17:33:01.199373 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.199247 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b374e83-1270-47c2-9620-88d9c947646d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4\" (UID: \"6b374e83-1270-47c2-9620-88d9c947646d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" Oct 30 17:33:01.200838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.200810 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd"] Oct 30 17:33:01.215002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.214967 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd"] Oct 30 17:33:01.215192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.215085 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" Oct 30 17:33:01.300452 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.300409 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzntf\" (UniqueName: \"kubernetes.io/projected/6b374e83-1270-47c2-9620-88d9c947646d-kube-api-access-zzntf\") pod \"payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4\" (UID: \"6b374e83-1270-47c2-9620-88d9c947646d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" Oct 30 17:33:01.300776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.300746 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b374e83-1270-47c2-9620-88d9c947646d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4\" (UID: \"6b374e83-1270-47c2-9620-88d9c947646d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" Oct 30 17:33:01.300989 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.300968 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e87a986-6278-48a7-b32e-742f7efd3c05-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd\" (UID: \"9e87a986-6278-48a7-b32e-742f7efd3c05\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" Oct 30 17:33:01.301201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.301181 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftlch\" (UniqueName: \"kubernetes.io/projected/9e87a986-6278-48a7-b32e-742f7efd3c05-kube-api-access-ftlch\") pod \"payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd\" (UID: \"9e87a986-6278-48a7-b32e-742f7efd3c05\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" Oct 30 17:33:01.301749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.301712 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b374e83-1270-47c2-9620-88d9c947646d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4\" (UID: \"6b374e83-1270-47c2-9620-88d9c947646d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" Oct 30 17:33:01.325774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.325727 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzntf\" (UniqueName: \"kubernetes.io/projected/6b374e83-1270-47c2-9620-88d9c947646d-kube-api-access-zzntf\") pod \"payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4\" (UID: \"6b374e83-1270-47c2-9620-88d9c947646d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" Oct 30 17:33:01.402708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.402616 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftlch\" (UniqueName: \"kubernetes.io/projected/9e87a986-6278-48a7-b32e-742f7efd3c05-kube-api-access-ftlch\") pod \"payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd\" (UID: \"9e87a986-6278-48a7-b32e-742f7efd3c05\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" Oct 30 17:33:01.403120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.402976 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e87a986-6278-48a7-b32e-742f7efd3c05-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd\" (UID: \"9e87a986-6278-48a7-b32e-742f7efd3c05\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" Oct 30 17:33:01.403982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.403958 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e87a986-6278-48a7-b32e-742f7efd3c05-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd\" (UID: \"9e87a986-6278-48a7-b32e-742f7efd3c05\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" Oct 30 17:33:01.427326 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.427286 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftlch\" (UniqueName: \"kubernetes.io/projected/9e87a986-6278-48a7-b32e-742f7efd3c05-kube-api-access-ftlch\") pod \"payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd\" (UID: \"9e87a986-6278-48a7-b32e-742f7efd3c05\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" Oct 30 17:33:01.519579 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.519540 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" Oct 30 17:33:01.544406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.544379 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" Oct 30 17:33:01.702290 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.702253 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:01.850629 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:33:01.850584 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b374e83_1270_47c2_9620_88d9c947646d.slice/crio-f5c302143902d7ffb54c19ca70987d24bd702d7fdb6c9e05930b2982e2832cf8 WatchSource:0}: Error finding container f5c302143902d7ffb54c19ca70987d24bd702d7fdb6c9e05930b2982e2832cf8: Status 404 returned error can't find the container with id f5c302143902d7ffb54c19ca70987d24bd702d7fdb6c9e05930b2982e2832cf8 Oct 30 17:33:01.852668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.852635 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4"] Oct 30 17:33:01.892630 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:01.892268 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd"] Oct 30 17:33:01.893521 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:33:01.893495 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e87a986_6278_48a7_b32e_742f7efd3c05.slice/crio-757f5181e64ecd7071c97c34d504eb6809a24b892c3aaee9343186cb12789d67 WatchSource:0}: Error finding container 757f5181e64ecd7071c97c34d504eb6809a24b892c3aaee9343186cb12789d67: Status 404 returned error can't find the container with id 757f5181e64ecd7071c97c34d504eb6809a24b892c3aaee9343186cb12789d67 Oct 30 17:33:02.228198 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:02.228098 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" event={"ID":"6b374e83-1270-47c2-9620-88d9c947646d","Type":"ContainerStarted","Data":"035e296a0ff89159590a3fc7a5cc958bea30fb90d29267afb3562d1f6a3f303b"} Oct 30 17:33:02.228198 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:02.228170 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" Oct 30 17:33:02.228401 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:02.228194 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" event={"ID":"6b374e83-1270-47c2-9620-88d9c947646d","Type":"ContainerStarted","Data":"f5c302143902d7ffb54c19ca70987d24bd702d7fdb6c9e05930b2982e2832cf8"} Oct 30 17:33:02.230859 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:02.230830 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" event={"ID":"9e87a986-6278-48a7-b32e-742f7efd3c05","Type":"ContainerStarted","Data":"ed7b593f8f96f5b7481e5043acf1076588ee4dd22b907a3734f8a8fa05676529"} Oct 30 17:33:02.231012 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:02.230882 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" event={"ID":"9e87a986-6278-48a7-b32e-742f7efd3c05","Type":"ContainerStarted","Data":"757f5181e64ecd7071c97c34d504eb6809a24b892c3aaee9343186cb12789d67"} Oct 30 17:33:02.231012 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:02.230923 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" Oct 30 17:33:02.295217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:02.295164 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" podStartSLOduration=2.295151113 podStartE2EDuration="2.295151113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:02.290941729 +0000 UTC m=+2759.936451907" watchObservedRunningTime="2026-10-30 17:33:02.295151113 +0000 UTC m=+2759.940661292" Oct 30 17:33:02.360445 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:02.360377 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" podStartSLOduration=1.360360863 podStartE2EDuration="1.360360863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:02.35609499 +0000 UTC m=+2760.001605171" watchObservedRunningTime="2026-10-30 17:33:02.360360863 +0000 UTC m=+2760.005871077" Oct 30 17:33:03.914922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:03.907391 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:04.066004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:04.065967 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:04.264768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:04.264169 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls"] Oct 30 17:33:04.264768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:04.264422 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" podUID="e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" containerName="istio-proxy" containerID="cri-o://bb865f51b48a7ef762794a788c6775c134649a3e260a7b56d96c5617d979660b" gracePeriod=30 Oct 30 17:33:04.266222 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:04.266192 4828 generic.go:356] "Generic (PLEG): container finished" podID="68e707eb-e7e5-4464-863d-89f0e71fe2a5" containerID="46b68ab8b5e66d80341a9a9c6f63ef94cc8e0f538f6025bc5ba1ba0b880bdb6e" exitCode=0 Oct 30 17:33:04.266463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:04.266328 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" event={"ID":"68e707eb-e7e5-4464-863d-89f0e71fe2a5","Type":"ContainerDied","Data":"46b68ab8b5e66d80341a9a9c6f63ef94cc8e0f538f6025bc5ba1ba0b880bdb6e"} Oct 30 17:33:04.319922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:04.319152 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" Oct 30 17:33:04.349992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:04.349928 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:04.443972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:04.441078 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68e707eb-e7e5-4464-863d-89f0e71fe2a5-kube-api-access-hj8fs\" (UniqueName: \"kubernetes.io/projected/68e707eb-e7e5-4464-863d-89f0e71fe2a5-kube-api-access-hj8fs\") pod \"68e707eb-e7e5-4464-863d-89f0e71fe2a5\" (UID: \"68e707eb-e7e5-4464-863d-89f0e71fe2a5\") " Oct 30 17:33:04.443972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:04.441223 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68e707eb-e7e5-4464-863d-89f0e71fe2a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e707eb-e7e5-4464-863d-89f0e71fe2a5-plugins-config-volume\") pod \"68e707eb-e7e5-4464-863d-89f0e71fe2a5\" (UID: \"68e707eb-e7e5-4464-863d-89f0e71fe2a5\") " Oct 30 17:33:04.443972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:04.442614 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e707eb-e7e5-4464-863d-89f0e71fe2a5-plugins-config-volume" pod "68e707eb-e7e5-4464-863d-89f0e71fe2a5" (UID: "68e707eb-e7e5-4464-863d-89f0e71fe2a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:04.479154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:04.477137 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e707eb-e7e5-4464-863d-89f0e71fe2a5-kube-api-access-hj8fs" pod "68e707eb-e7e5-4464-863d-89f0e71fe2a5" (UID: "68e707eb-e7e5-4464-863d-89f0e71fe2a5"). InnerVolumeSpecName "kube-api-access-hj8fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:04.559338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:04.557272 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj8fs\" (UniqueName: \"kubernetes.io/projected/68e707eb-e7e5-4464-863d-89f0e71fe2a5-kube-api-access-hj8fs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:04.559338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:04.557313 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68e707eb-e7e5-4464-863d-89f0e71fe2a5-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:05.280547 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:05.280519 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" Oct 30 17:33:05.280547 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:05.280536 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw" event={"ID":"68e707eb-e7e5-4464-863d-89f0e71fe2a5","Type":"ContainerDied","Data":"759dee3b64d49747d81edfd97271054be66e2e4b7d7252f86276b4652b6907b5"} Oct 30 17:33:05.281146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:05.280603 4828 scope.go:122] "RemoveContainer" containerID="46b68ab8b5e66d80341a9a9c6f63ef94cc8e0f538f6025bc5ba1ba0b880bdb6e" Oct 30 17:33:05.345488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:05.345453 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw"] Oct 30 17:33:05.357778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:05.357746 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-987dfc-6999b6d55d-z2cxw"] Oct 30 17:33:05.590019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:05.589554 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:06.493658 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:06.493623 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68e707eb-e7e5-4464-863d-89f0e71fe2a5" path="/var/lib/kubelet/pods/68e707eb-e7e5-4464-863d-89f0e71fe2a5/volumes" Oct 30 17:33:07.179937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.179865 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv"] Oct 30 17:33:07.180829 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.180805 4828 cpu_manager.go:414] "containerMap: removing container" podUID="68e707eb-e7e5-4464-863d-89f0e71fe2a5" containerName="payload-processing" Oct 30 17:33:07.180829 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.180828 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e707eb-e7e5-4464-863d-89f0e71fe2a5" containerName="payload-processing" Oct 30 17:33:07.181129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.181112 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e707eb-e7e5-4464-863d-89f0e71fe2a5" containerName="payload-processing" Oct 30 17:33:07.186368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.186347 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.209219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.209192 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv"] Oct 30 17:33:07.213779 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.213264 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a949e7-serving-cert\"" Oct 30 17:33:07.215853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.215829 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264f7990-6cc2-4018-a1c5-cc76ccfb5338-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.215995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.215922 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjghp\" (UniqueName: \"kubernetes.io/projected/264f7990-6cc2-4018-a1c5-cc76ccfb5338-kube-api-access-xjghp\") pod \"maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.216096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.216071 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-metrics-certs\") pod \"maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.216295 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.216269 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.297860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.297805 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:07.318209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.318165 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.318403 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:33:07.318303 4828 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-serving-cert: secret "maas-api-e2e-ipp-a-a949e7-serving-cert" not found Oct 30 17:33:07.318403 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.318338 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264f7990-6cc2-4018-a1c5-cc76ccfb5338-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.318403 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:33:07.318361 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-maas-api-tls podName:264f7990-6cc2-4018-a1c5-cc76ccfb5338 nodeName:}" failed. No retries permitted until 2026-10-30 17:33:07.818346829 +0000 UTC m=+2765.463857004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-maas-api-tls") pod "maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" (UID: "264f7990-6cc2-4018-a1c5-cc76ccfb5338") : secret "maas-api-e2e-ipp-a-a949e7-serving-cert" not found Oct 30 17:33:07.318598 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.318436 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjghp\" (UniqueName: \"kubernetes.io/projected/264f7990-6cc2-4018-a1c5-cc76ccfb5338-kube-api-access-xjghp\") pod \"maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.318598 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.318534 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-metrics-certs\") pod \"maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.319051 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.319028 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264f7990-6cc2-4018-a1c5-cc76ccfb5338-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.321322 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.321297 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-metrics-certs\") pod \"maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.369948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.358625 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjghp\" (UniqueName: \"kubernetes.io/projected/264f7990-6cc2-4018-a1c5-cc76ccfb5338-kube-api-access-xjghp\") pod \"maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.465038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.464951 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:07.624595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.624554 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:07.828707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.828668 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.831380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:07.831345 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:07.913091 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:33:07.913052 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:33:08.107566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:08.107472 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:08.420661 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:33:08.420630 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264f7990_6cc2_4018_a1c5_cc76ccfb5338.slice/crio-9276dedd8be531f8a8e52d98d0291228c5981681b360ed373a042f7ac06848b1 WatchSource:0}: Error finding container 9276dedd8be531f8a8e52d98d0291228c5981681b360ed373a042f7ac06848b1: Status 404 returned error can't find the container with id 9276dedd8be531f8a8e52d98d0291228c5981681b360ed373a042f7ac06848b1 Oct 30 17:33:08.421725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:08.421697 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv"] Oct 30 17:33:08.446538 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:08.446489 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:33:08.446708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:08.446551 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" podUID="e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:33:09.244151 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.216423 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:09.325917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.325861 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" event={"ID":"264f7990-6cc2-4018-a1c5-cc76ccfb5338","Type":"ContainerStarted","Data":"d201b3ffcb636acb0b884de8024cbc3357a050e852623ce5ba1c4ecdc95077c7"} Oct 30 17:33:09.328962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.327123 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:09.328962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.327164 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" event={"ID":"264f7990-6cc2-4018-a1c5-cc76ccfb5338","Type":"ContainerStarted","Data":"9276dedd8be531f8a8e52d98d0291228c5981681b360ed373a042f7ac06848b1"} Oct 30 17:33:09.328962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.327798 4828 generic.go:356] "Generic (PLEG): container finished" podID="e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" containerID="bb865f51b48a7ef762794a788c6775c134649a3e260a7b56d96c5617d979660b" exitCode=0 Oct 30 17:33:09.328962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.328057 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" event={"ID":"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee","Type":"ContainerDied","Data":"bb865f51b48a7ef762794a788c6775c134649a3e260a7b56d96c5617d979660b"} Oct 30 17:33:09.563252 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.558816 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:33:09.605995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.605931 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" podStartSLOduration=2.151230546 podStartE2EDuration="2.605912819s" totalImagesPullingTime="454.682273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:33:09.384937143 +0000 UTC m=+2767.030447324" watchObservedRunningTime="2026-10-30 17:33:09.605912819 +0000 UTC m=+2767.251422991" Oct 30 17:33:09.650984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.650893 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-podinfo\") pod \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " Oct 30 17:33:09.651166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.651041 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-token\" (UniqueName: \"kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-token\") pod \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " Oct 30 17:33:09.651166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.651097 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istiod-ca-cert\") pod \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " Oct 30 17:33:09.651312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.651164 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-socket\") pod \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " Oct 30 17:33:09.651312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.651229 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-certs\") pod \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " Oct 30 17:33:09.651312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.651282 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-envoy\") pod \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " Oct 30 17:33:09.651493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.651370 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-data\") pod \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " Oct 30 17:33:09.651493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.651417 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-credential-socket\") pod \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " Oct 30 17:33:09.651493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.651477 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-kube-api-access-qx9k6\" (UniqueName: \"kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-kube-api-access-qx9k6\") pod \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\" (UID: \"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee\") " Oct 30 17:33:09.654147 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.654107 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-socket" pod "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" (UID: "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:09.658996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.654835 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-credential-socket" pod "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" (UID: "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:09.659150 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.654965 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-certs" pod "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" (UID: "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:09.659150 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.655153 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-data" pod "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" (UID: "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:09.659351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.656569 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istiod-ca-cert" pod "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" (UID: "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:09.659547 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.659433 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-token" pod "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" (UID: "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:09.665755 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.665718 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-kube-api-access-qx9k6" pod "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" (UID: "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee"). InnerVolumeSpecName "kube-api-access-qx9k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:09.666624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.666473 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-podinfo" pod "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" (UID: "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:33:09.670353 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.670320 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-envoy" pod "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" (UID: "e49de0e3-8b4f-4f8a-b14c-aa94892f6dee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:09.754136 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.754102 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:09.754136 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.754132 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:09.754136 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.754142 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:09.754380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.754150 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:09.754380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.754161 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:09.754380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.754169 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:09.754380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.754176 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:09.754380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.754186 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:09.754380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:09.754194 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx9k6\" (UniqueName: \"kubernetes.io/projected/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee-kube-api-access-qx9k6\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:10.337870 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:10.337832 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" Oct 30 17:33:10.338339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:10.337838 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls" event={"ID":"e49de0e3-8b4f-4f8a-b14c-aa94892f6dee","Type":"ContainerDied","Data":"027b9090afe866bd27977540175d3a05db55d514e8617f70e4c2b3b5bdba4d27"} Oct 30 17:33:10.338339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:10.338020 4828 scope.go:122] "RemoveContainer" containerID="bb865f51b48a7ef762794a788c6775c134649a3e260a7b56d96c5617d979660b" Oct 30 17:33:10.408115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:10.408078 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls"] Oct 30 17:33:10.421158 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:10.421121 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-987dfc-openshift-default-bd796698f-szbls"] Oct 30 17:33:10.496586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:10.496541 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" path="/var/lib/kubelet/pods/e49de0e3-8b4f-4f8a-b14c-aa94892f6dee/volumes" Oct 30 17:33:10.948825 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:10.948785 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:13.183435 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:13.183386 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:13.255341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:13.255296 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" Oct 30 17:33:13.255826 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:13.255799 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" Oct 30 17:33:15.110316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:15.110271 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:15.294312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:15.294269 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:15.344319 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:15.344290 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:33:15.492711 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:15.492609 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:16.985025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:16.984986 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5"] Oct 30 17:33:16.985669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:16.985625 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" podUID="694fc7b7-12ff-4f40-a513-44fd85f92439" containerName="maas-api" containerID="cri-o://b768bbb928cd64c9c04771d934a87bc520c6c8a19044ae78ee71e759ef0821f5" gracePeriod=30 Oct 30 17:33:17.104934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.104885 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:17.383221 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.383184 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:33:17.522549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.521379 4828 generic.go:356] "Generic (PLEG): container finished" podID="694fc7b7-12ff-4f40-a513-44fd85f92439" containerID="b768bbb928cd64c9c04771d934a87bc520c6c8a19044ae78ee71e759ef0821f5" exitCode=0 Oct 30 17:33:17.522549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.521597 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" event={"ID":"694fc7b7-12ff-4f40-a513-44fd85f92439","Type":"ContainerDied","Data":"b768bbb928cd64c9c04771d934a87bc520c6c8a19044ae78ee71e759ef0821f5"} Oct 30 17:33:17.522549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.521655 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" event={"ID":"694fc7b7-12ff-4f40-a513-44fd85f92439","Type":"ContainerDied","Data":"3fa6e41a294fb89a794a6b481feb2e593e0522665722b9584cd42e8e16c1073f"} Oct 30 17:33:17.522549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.521686 4828 scope.go:122] "RemoveContainer" containerID="b768bbb928cd64c9c04771d934a87bc520c6c8a19044ae78ee71e759ef0821f5" Oct 30 17:33:17.522549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.521858 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5" Oct 30 17:33:17.543725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.543696 4828 scope.go:122] "RemoveContainer" containerID="b768bbb928cd64c9c04771d934a87bc520c6c8a19044ae78ee71e759ef0821f5" Oct 30 17:33:17.544160 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:33:17.544128 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b768bbb928cd64c9c04771d934a87bc520c6c8a19044ae78ee71e759ef0821f5\": container with ID starting with b768bbb928cd64c9c04771d934a87bc520c6c8a19044ae78ee71e759ef0821f5 not found: ID does not exist" containerID="b768bbb928cd64c9c04771d934a87bc520c6c8a19044ae78ee71e759ef0821f5" Oct 30 17:33:17.544291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.544173 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b768bbb928cd64c9c04771d934a87bc520c6c8a19044ae78ee71e759ef0821f5"} err="failed to get container status \"b768bbb928cd64c9c04771d934a87bc520c6c8a19044ae78ee71e759ef0821f5\": rpc error: code = NotFound desc = could not find container \"b768bbb928cd64c9c04771d934a87bc520c6c8a19044ae78ee71e759ef0821f5\": container with ID starting with b768bbb928cd64c9c04771d934a87bc520c6c8a19044ae78ee71e759ef0821f5 not found: ID does not exist" Oct 30 17:33:17.566116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.566079 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-maas-api-tls\") pod \"694fc7b7-12ff-4f40-a513-44fd85f92439\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " Oct 30 17:33:17.566309 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.566178 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-metrics-certs\" (UniqueName: \"kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-metrics-certs\") pod \"694fc7b7-12ff-4f40-a513-44fd85f92439\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " Oct 30 17:33:17.566422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.566355 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/694fc7b7-12ff-4f40-a513-44fd85f92439-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/694fc7b7-12ff-4f40-a513-44fd85f92439-openshift-service-ca\") pod \"694fc7b7-12ff-4f40-a513-44fd85f92439\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " Oct 30 17:33:17.566559 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.566519 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694fc7b7-12ff-4f40-a513-44fd85f92439-kube-api-access-mjzcj\" (UniqueName: \"kubernetes.io/projected/694fc7b7-12ff-4f40-a513-44fd85f92439-kube-api-access-mjzcj\") pod \"694fc7b7-12ff-4f40-a513-44fd85f92439\" (UID: \"694fc7b7-12ff-4f40-a513-44fd85f92439\") " Oct 30 17:33:17.568634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.568602 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694fc7b7-12ff-4f40-a513-44fd85f92439-openshift-service-ca" pod "694fc7b7-12ff-4f40-a513-44fd85f92439" (UID: "694fc7b7-12ff-4f40-a513-44fd85f92439"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:17.577490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.577458 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694fc7b7-12ff-4f40-a513-44fd85f92439-kube-api-access-mjzcj" pod "694fc7b7-12ff-4f40-a513-44fd85f92439" (UID: "694fc7b7-12ff-4f40-a513-44fd85f92439"). InnerVolumeSpecName "kube-api-access-mjzcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:17.585106 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.585053 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-maas-api-tls" pod "694fc7b7-12ff-4f40-a513-44fd85f92439" (UID: "694fc7b7-12ff-4f40-a513-44fd85f92439"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:33:17.585696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.585673 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-metrics-certs" pod "694fc7b7-12ff-4f40-a513-44fd85f92439" (UID: "694fc7b7-12ff-4f40-a513-44fd85f92439"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:33:17.760926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.687421 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjzcj\" (UniqueName: \"kubernetes.io/projected/694fc7b7-12ff-4f40-a513-44fd85f92439-kube-api-access-mjzcj\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:17.760926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.687462 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:17.760926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.687478 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/694fc7b7-12ff-4f40-a513-44fd85f92439-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:17.760926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.687493 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/694fc7b7-12ff-4f40-a513-44fd85f92439-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:17.937044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.936987 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5"] Oct 30 17:33:17.945435 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:17.945357 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-001a96-5fdc67c77-zxxp5"] Oct 30 17:33:18.056884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:18.056438 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:18.248880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:18.242561 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:18.493707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:18.493670 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="694fc7b7-12ff-4f40-a513-44fd85f92439" path="/var/lib/kubelet/pods/694fc7b7-12ff-4f40-a513-44fd85f92439/volumes" Oct 30 17:33:18.698061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:18.697999 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:20.135117 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:20.135075 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:21.891502 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:21.891465 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:23.623554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:23.623023 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:25.240542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:25.240491 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:25.415637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:25.415592 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:25.564039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:25.564001 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:25.743942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:25.743889 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:26.103643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:26.103605 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:27.469963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:27.463094 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:29.267272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:29.267233 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:31.209965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:31.207532 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:31.513982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:31.511583 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:32.871518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:32.871476 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:33.028490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:33.028449 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:33.900107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:33.900054 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:35.258981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:35.258947 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf"] Oct 30 17:33:35.446488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:35.446437 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:35.596572 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:35.596479 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:35.751892 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:35.751860 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:37.225830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:37.225794 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:37.377438 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:37.377403 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:37.530490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:37.530456 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:37.879675 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:37.879582 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp"] Oct 30 17:33:37.887143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:37.880121 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" podUID="fac5920b-1b21-4328-ae7f-4586183f34c7" containerName="istio-proxy" containerID="cri-o://51f8767de44343fa43926c0ae7b6cf86e074b039f70332a705549af43e92a034" gracePeriod=30 Oct 30 17:33:38.023109 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:33:38.023032 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:33:38.803363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:38.803314 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:38.990679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:38.990639 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:39.145149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:39.145055 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:39.481463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:39.481374 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:39.640928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:39.640874 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:39.728458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:39.728419 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf"] Oct 30 17:33:39.728643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:39.728623 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" containerID="cri-o://7209be06affa8802f4fe4e841c4390ef0b92b08532918c50da04d2b1b5faf541" gracePeriod=30 Oct 30 17:33:40.171846 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.171803 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:40.259581 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.259537 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" podUID="785fe8ec-352e-4c68-ad24-924106f4dc53" containerName="payload-processing" containerID="cri-o://c1d1e11dbef03a81336c7b4426374eb0cc7b9d61ecd7df08712be60bafe34eaa" gracePeriod=25 Oct 30 17:33:40.565186 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.565153 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" Oct 30 17:33:40.714642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.714551 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/785fe8ec-352e-4c68-ad24-924106f4dc53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785fe8ec-352e-4c68-ad24-924106f4dc53-plugins-config-volume\") pod \"785fe8ec-352e-4c68-ad24-924106f4dc53\" (UID: \"785fe8ec-352e-4c68-ad24-924106f4dc53\") " Oct 30 17:33:40.714642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.714634 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/785fe8ec-352e-4c68-ad24-924106f4dc53-kube-api-access-k9p5s\" (UniqueName: \"kubernetes.io/projected/785fe8ec-352e-4c68-ad24-924106f4dc53-kube-api-access-k9p5s\") pod \"785fe8ec-352e-4c68-ad24-924106f4dc53\" (UID: \"785fe8ec-352e-4c68-ad24-924106f4dc53\") " Oct 30 17:33:40.715064 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.715042 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/785fe8ec-352e-4c68-ad24-924106f4dc53-plugins-config-volume" pod "785fe8ec-352e-4c68-ad24-924106f4dc53" (UID: "785fe8ec-352e-4c68-ad24-924106f4dc53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:40.715370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.715351 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/785fe8ec-352e-4c68-ad24-924106f4dc53-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:40.717271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.717242 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785fe8ec-352e-4c68-ad24-924106f4dc53-kube-api-access-k9p5s" pod "785fe8ec-352e-4c68-ad24-924106f4dc53" (UID: "785fe8ec-352e-4c68-ad24-924106f4dc53"). InnerVolumeSpecName "kube-api-access-k9p5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:40.817485 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.817444 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9p5s\" (UniqueName: \"kubernetes.io/projected/785fe8ec-352e-4c68-ad24-924106f4dc53-kube-api-access-k9p5s\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:40.945026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.944987 4828 generic.go:356] "Generic (PLEG): container finished" podID="785fe8ec-352e-4c68-ad24-924106f4dc53" containerID="c1d1e11dbef03a81336c7b4426374eb0cc7b9d61ecd7df08712be60bafe34eaa" exitCode=0 Oct 30 17:33:40.945213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.945033 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" Oct 30 17:33:40.945213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.945071 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" event={"ID":"785fe8ec-352e-4c68-ad24-924106f4dc53","Type":"ContainerDied","Data":"c1d1e11dbef03a81336c7b4426374eb0cc7b9d61ecd7df08712be60bafe34eaa"} Oct 30 17:33:40.945213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.945122 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf" event={"ID":"785fe8ec-352e-4c68-ad24-924106f4dc53","Type":"ContainerDied","Data":"00f1cbc40be038280d19270a117d092efe50c00c441f3adcba59566c48d63cb2"} Oct 30 17:33:40.945213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.945145 4828 scope.go:122] "RemoveContainer" containerID="c1d1e11dbef03a81336c7b4426374eb0cc7b9d61ecd7df08712be60bafe34eaa" Oct 30 17:33:40.956235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.956205 4828 scope.go:122] "RemoveContainer" containerID="c1d1e11dbef03a81336c7b4426374eb0cc7b9d61ecd7df08712be60bafe34eaa" Oct 30 17:33:40.956517 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:33:40.956499 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d1e11dbef03a81336c7b4426374eb0cc7b9d61ecd7df08712be60bafe34eaa\": container with ID starting with c1d1e11dbef03a81336c7b4426374eb0cc7b9d61ecd7df08712be60bafe34eaa not found: ID does not exist" containerID="c1d1e11dbef03a81336c7b4426374eb0cc7b9d61ecd7df08712be60bafe34eaa" Oct 30 17:33:40.956566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:40.956526 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d1e11dbef03a81336c7b4426374eb0cc7b9d61ecd7df08712be60bafe34eaa"} err="failed to get container status \"c1d1e11dbef03a81336c7b4426374eb0cc7b9d61ecd7df08712be60bafe34eaa\": rpc error: code = NotFound desc = could not find container \"c1d1e11dbef03a81336c7b4426374eb0cc7b9d61ecd7df08712be60bafe34eaa\": container with ID starting with c1d1e11dbef03a81336c7b4426374eb0cc7b9d61ecd7df08712be60bafe34eaa not found: ID does not exist" Oct 30 17:33:41.001860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:41.001768 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf"] Oct 30 17:33:41.013820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:41.013785 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4fb821-647d8877b-dw5nf"] Oct 30 17:33:41.245964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:41.245924 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:41.441372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:41.441334 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:41.613925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:41.592056 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:41.675924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:41.669936 4828 patch_prober.go:29] interesting pod/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:33:41.675924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:41.670006 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:33:42.020774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:42.020738 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:42.235629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:42.235584 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:42.389218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:42.389125 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:42.497949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:42.497888 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="785fe8ec-352e-4c68-ad24-924106f4dc53" path="/var/lib/kubelet/pods/785fe8ec-352e-4c68-ad24-924106f4dc53/volumes" Oct 30 17:33:42.769971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:42.769872 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:43.325235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.325207 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:33:43.439271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.439236 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-envoy\") pod \"fac5920b-1b21-4328-ae7f-4586183f34c7\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " Oct 30 17:33:43.439800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.439359 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-podinfo\") pod \"fac5920b-1b21-4328-ae7f-4586183f34c7\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " Oct 30 17:33:43.439800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.439521 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-token\" (UniqueName: \"kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-token\") pod \"fac5920b-1b21-4328-ae7f-4586183f34c7\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " Oct 30 17:33:43.439800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.439577 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-socket\") pod \"fac5920b-1b21-4328-ae7f-4586183f34c7\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " Oct 30 17:33:43.439800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.439608 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-kube-api-access-qknmv\" (UniqueName: \"kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-kube-api-access-qknmv\") pod \"fac5920b-1b21-4328-ae7f-4586183f34c7\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " Oct 30 17:33:43.440381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.439938 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-socket" pod "fac5920b-1b21-4328-ae7f-4586183f34c7" (UID: "fac5920b-1b21-4328-ae7f-4586183f34c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:43.445103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.440502 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac5920b-1b21-4328-ae7f-4586183f34c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac5920b-1b21-4328-ae7f-4586183f34c7-istiod-ca-cert\") pod \"fac5920b-1b21-4328-ae7f-4586183f34c7\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " Oct 30 17:33:43.445103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.440543 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac5920b-1b21-4328-ae7f-4586183f34c7-istiod-ca-cert" pod "fac5920b-1b21-4328-ae7f-4586183f34c7" (UID: "fac5920b-1b21-4328-ae7f-4586183f34c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:43.445103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.440608 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-certs\") pod \"fac5920b-1b21-4328-ae7f-4586183f34c7\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " Oct 30 17:33:43.445103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.440793 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-credential-socket\") pod \"fac5920b-1b21-4328-ae7f-4586183f34c7\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " Oct 30 17:33:43.445103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.440879 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-data\") pod \"fac5920b-1b21-4328-ae7f-4586183f34c7\" (UID: \"fac5920b-1b21-4328-ae7f-4586183f34c7\") " Oct 30 17:33:43.445103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.440914 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-certs" pod "fac5920b-1b21-4328-ae7f-4586183f34c7" (UID: "fac5920b-1b21-4328-ae7f-4586183f34c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:43.445103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.441103 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-credential-socket" pod "fac5920b-1b21-4328-ae7f-4586183f34c7" (UID: "fac5920b-1b21-4328-ae7f-4586183f34c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:43.445103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.441283 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-data" pod "fac5920b-1b21-4328-ae7f-4586183f34c7" (UID: "fac5920b-1b21-4328-ae7f-4586183f34c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:43.445103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.442311 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-podinfo" pod "fac5920b-1b21-4328-ae7f-4586183f34c7" (UID: "fac5920b-1b21-4328-ae7f-4586183f34c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:33:43.445103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.442404 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-envoy" pod "fac5920b-1b21-4328-ae7f-4586183f34c7" (UID: "fac5920b-1b21-4328-ae7f-4586183f34c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:43.445103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.442660 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-token" pod "fac5920b-1b21-4328-ae7f-4586183f34c7" (UID: "fac5920b-1b21-4328-ae7f-4586183f34c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:43.445103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.443716 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-kube-api-access-qknmv" pod "fac5920b-1b21-4328-ae7f-4586183f34c7" (UID: "fac5920b-1b21-4328-ae7f-4586183f34c7"). InnerVolumeSpecName "kube-api-access-qknmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:43.445973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.445951 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:43.446079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.445975 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:43.446079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.445989 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:43.446079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.445999 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:43.446079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.446014 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:43.446079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.446027 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qknmv\" (UniqueName: \"kubernetes.io/projected/fac5920b-1b21-4328-ae7f-4586183f34c7-kube-api-access-qknmv\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:43.446079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.446042 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac5920b-1b21-4328-ae7f-4586183f34c7-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:43.446079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.446058 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:43.446079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.446070 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5920b-1b21-4328-ae7f-4586183f34c7-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:43.975763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.975679 4828 generic.go:356] "Generic (PLEG): container finished" podID="fac5920b-1b21-4328-ae7f-4586183f34c7" containerID="51f8767de44343fa43926c0ae7b6cf86e074b039f70332a705549af43e92a034" exitCode=0 Oct 30 17:33:43.976008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.975778 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" Oct 30 17:33:43.976393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.975725 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" event={"ID":"fac5920b-1b21-4328-ae7f-4586183f34c7","Type":"ContainerDied","Data":"51f8767de44343fa43926c0ae7b6cf86e074b039f70332a705549af43e92a034"} Oct 30 17:33:43.976393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.976164 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp" event={"ID":"fac5920b-1b21-4328-ae7f-4586183f34c7","Type":"ContainerDied","Data":"af60dd386fffd1ba03ccb9424e2e657e4a937f9b60e0cefabb30bc6fbb1f15ea"} Oct 30 17:33:43.976393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.976195 4828 scope.go:122] "RemoveContainer" containerID="51f8767de44343fa43926c0ae7b6cf86e074b039f70332a705549af43e92a034" Oct 30 17:33:43.993396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.993370 4828 scope.go:122] "RemoveContainer" containerID="51f8767de44343fa43926c0ae7b6cf86e074b039f70332a705549af43e92a034" Oct 30 17:33:43.993748 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:33:43.993714 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f8767de44343fa43926c0ae7b6cf86e074b039f70332a705549af43e92a034\": container with ID starting with 51f8767de44343fa43926c0ae7b6cf86e074b039f70332a705549af43e92a034 not found: ID does not exist" containerID="51f8767de44343fa43926c0ae7b6cf86e074b039f70332a705549af43e92a034" Oct 30 17:33:43.993865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:43.993760 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f8767de44343fa43926c0ae7b6cf86e074b039f70332a705549af43e92a034"} err="failed to get container status \"51f8767de44343fa43926c0ae7b6cf86e074b039f70332a705549af43e92a034\": rpc error: code = NotFound desc = could not find container \"51f8767de44343fa43926c0ae7b6cf86e074b039f70332a705549af43e92a034\": container with ID starting with 51f8767de44343fa43926c0ae7b6cf86e074b039f70332a705549af43e92a034 not found: ID does not exist" Oct 30 17:33:44.051837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:44.051807 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp"] Oct 30 17:33:44.089094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:44.082365 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5d8947-openshift-default-79cb6c4579-kpkfp"] Oct 30 17:33:44.096924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:44.092469 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:44.494352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:44.494312 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac5920b-1b21-4328-ae7f-4586183f34c7" path="/var/lib/kubelet/pods/fac5920b-1b21-4328-ae7f-4586183f34c7/volumes" Oct 30 17:33:44.990289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:44.990255 4828 generic.go:356] "Generic (PLEG): container finished" podID="15f58c09-e14a-41e4-a355-c08c7575c648" containerID="7209be06affa8802f4fe4e841c4390ef0b92b08532918c50da04d2b1b5faf541" exitCode=0 Oct 30 17:33:44.990466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:44.990315 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" event={"ID":"15f58c09-e14a-41e4-a355-c08c7575c648","Type":"ContainerDied","Data":"7209be06affa8802f4fe4e841c4390ef0b92b08532918c50da04d2b1b5faf541"} Oct 30 17:33:44.990466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:44.990390 4828 scope.go:122] "RemoveContainer" containerID="3eecf2d5680e61f434ed7d8651f875ace7b03a6030cb64842a3dbf097a10981f" Oct 30 17:33:45.034731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.034703 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:33:45.191806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.191766 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-socket\") pod \"15f58c09-e14a-41e4-a355-c08c7575c648\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " Oct 30 17:33:45.192043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.191875 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15f58c09-e14a-41e4-a355-c08c7575c648-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f58c09-e14a-41e4-a355-c08c7575c648-istio-podinfo\") pod \"15f58c09-e14a-41e4-a355-c08c7575c648\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " Oct 30 17:33:45.192043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.191944 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-kube-api-access-74mv7\" (UniqueName: \"kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-kube-api-access-74mv7\") pod \"15f58c09-e14a-41e4-a355-c08c7575c648\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " Oct 30 17:33:45.192175 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.192114 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-socket" pod "15f58c09-e14a-41e4-a355-c08c7575c648" (UID: "15f58c09-e14a-41e4-a355-c08c7575c648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:45.192235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.192174 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f58c09-e14a-41e4-a355-c08c7575c648-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f58c09-e14a-41e4-a355-c08c7575c648-istiod-ca-cert\") pod \"15f58c09-e14a-41e4-a355-c08c7575c648\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " Oct 30 17:33:45.192235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.192218 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-data\") pod \"15f58c09-e14a-41e4-a355-c08c7575c648\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " Oct 30 17:33:45.192353 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.192294 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-credential-socket\") pod \"15f58c09-e14a-41e4-a355-c08c7575c648\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " Oct 30 17:33:45.192488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.192423 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-envoy\") pod \"15f58c09-e14a-41e4-a355-c08c7575c648\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " Oct 30 17:33:45.193640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.192653 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-certs\") pod \"15f58c09-e14a-41e4-a355-c08c7575c648\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " Oct 30 17:33:45.193640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.192719 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-istio-token\" (UniqueName: \"kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-istio-token\") pod \"15f58c09-e14a-41e4-a355-c08c7575c648\" (UID: \"15f58c09-e14a-41e4-a355-c08c7575c648\") " Oct 30 17:33:45.193640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.192918 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f58c09-e14a-41e4-a355-c08c7575c648-istiod-ca-cert" pod "15f58c09-e14a-41e4-a355-c08c7575c648" (UID: "15f58c09-e14a-41e4-a355-c08c7575c648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:45.193640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.193334 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-data" pod "15f58c09-e14a-41e4-a355-c08c7575c648" (UID: "15f58c09-e14a-41e4-a355-c08c7575c648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:45.193640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.193451 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-credential-socket" pod "15f58c09-e14a-41e4-a355-c08c7575c648" (UID: "15f58c09-e14a-41e4-a355-c08c7575c648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:45.194018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.193748 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:45.194018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.193774 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f58c09-e14a-41e4-a355-c08c7575c648-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:45.194018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.193790 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:45.194018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.193803 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:45.194018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.193867 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-certs" pod "15f58c09-e14a-41e4-a355-c08c7575c648" (UID: "15f58c09-e14a-41e4-a355-c08c7575c648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:45.194658 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.194633 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-kube-api-access-74mv7" pod "15f58c09-e14a-41e4-a355-c08c7575c648" (UID: "15f58c09-e14a-41e4-a355-c08c7575c648"). InnerVolumeSpecName "kube-api-access-74mv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:45.195338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.195314 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15f58c09-e14a-41e4-a355-c08c7575c648-istio-podinfo" pod "15f58c09-e14a-41e4-a355-c08c7575c648" (UID: "15f58c09-e14a-41e4-a355-c08c7575c648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:33:45.196028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.195982 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-envoy" pod "15f58c09-e14a-41e4-a355-c08c7575c648" (UID: "15f58c09-e14a-41e4-a355-c08c7575c648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:45.196122 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.196086 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-istio-token" pod "15f58c09-e14a-41e4-a355-c08c7575c648" (UID: "15f58c09-e14a-41e4-a355-c08c7575c648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:45.295883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.295793 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:45.295883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.295838 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f58c09-e14a-41e4-a355-c08c7575c648-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:45.295883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.295852 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:45.295883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.295865 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f58c09-e14a-41e4-a355-c08c7575c648-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:45.295883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:45.295879 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74mv7\" (UniqueName: \"kubernetes.io/projected/15f58c09-e14a-41e4-a355-c08c7575c648-kube-api-access-74mv7\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:46.017503 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:46.014698 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:46.053588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:46.053551 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" event={"ID":"15f58c09-e14a-41e4-a355-c08c7575c648","Type":"ContainerDied","Data":"dc30a213614c6697e52b54aa4df719cca7efb32573af4f32fd207b483ca8093f"} Oct 30 17:33:46.053889 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:46.053870 4828 scope.go:122] "RemoveContainer" containerID="7209be06affa8802f4fe4e841c4390ef0b92b08532918c50da04d2b1b5faf541" Oct 30 17:33:46.054178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:46.053626 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf" Oct 30 17:33:46.159495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:46.158877 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf"] Oct 30 17:33:46.163616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:46.163589 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-78c9c2-openshift-default-766fb66f65-k85gf"] Oct 30 17:33:46.507748 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:46.507681 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" path="/var/lib/kubelet/pods/15f58c09-e14a-41e4-a355-c08c7575c648/volumes" Oct 30 17:33:46.708545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:46.708504 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:46.856165 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:46.856129 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:47.002781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:47.002743 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4"] Oct 30 17:33:47.003059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:47.003027 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" containerID="cri-o://370b6afdec1d5c13505e239690ae5899fb4404b6c66d1fb5a0b48957ca7c4eb9" gracePeriod=30 Oct 30 17:33:47.131984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:47.122761 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:47.355707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:47.354021 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:47.549212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:47.549056 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:48.572924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:48.572499 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:49.540795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:49.540756 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:49.689082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:49.689042 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:49.818769 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:49.818675 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d"] Oct 30 17:33:49.819176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:49.818969 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" containerID="cri-o://072c9e8f1dcdee248f6e30898e5cf376293bc831b809f366e0a0b3843830a76a" gracePeriod=30 Oct 30 17:33:49.887593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:49.887543 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:50.082090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:50.081998 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:50.233197 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:50.233155 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:50.386468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:50.386339 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:52.195649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.195562 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:52.204177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.199190 4828 generic.go:356] "Generic (PLEG): container finished" podID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerID="370b6afdec1d5c13505e239690ae5899fb4404b6c66d1fb5a0b48957ca7c4eb9" exitCode=0 Oct 30 17:33:52.204177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.199473 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" event={"ID":"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9","Type":"ContainerDied","Data":"370b6afdec1d5c13505e239690ae5899fb4404b6c66d1fb5a0b48957ca7c4eb9"} Oct 30 17:33:52.204177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.199542 4828 scope.go:122] "RemoveContainer" containerID="458d56cc284942d9a9c51f0ec32659d5876c43e00048d9eccec47281518d91d3" Oct 30 17:33:52.356814 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.356787 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:33:52.390125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.390086 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-credential-socket\") pod \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " Oct 30 17:33:52.390125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.390137 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-socket\") pod \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " Oct 30 17:33:52.390396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.390203 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istiod-ca-cert\") pod \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " Oct 30 17:33:52.390396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.390313 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-podinfo\") pod \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " Oct 30 17:33:52.390396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.390383 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-certs\") pod \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " Oct 30 17:33:52.390580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.390425 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-credential-socket" pod "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" (UID: "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:52.390580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.390472 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-kube-api-access-f9thx\" (UniqueName: \"kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-kube-api-access-f9thx\") pod \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " Oct 30 17:33:52.390580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.390511 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-token\" (UniqueName: \"kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-token\") pod \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " Oct 30 17:33:52.390746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.390717 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-data\") pod \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " Oct 30 17:33:52.390810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.390734 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-certs" pod "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" (UID: "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:52.390810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.390764 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-envoy\") pod \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\" (UID: \"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9\") " Oct 30 17:33:52.390987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.390965 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-data" pod "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" (UID: "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:52.391237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.391211 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-socket" pod "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" (UID: "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:52.391465 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.391437 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istiod-ca-cert" pod "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" (UID: "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:52.391567 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.391545 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:52.391627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.391571 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:52.391627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.391589 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:52.391627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.391602 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:52.393591 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.393564 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-envoy" pod "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" (UID: "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:52.396334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.396289 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-podinfo" pod "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" (UID: "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:33:52.396502 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.396480 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-kube-api-access-f9thx" pod "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" (UID: "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9"). InnerVolumeSpecName "kube-api-access-f9thx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:52.407080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.407050 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-token" pod "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" (UID: "6c9edd23-7c40-4ee3-aa06-d68cc6800ab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:52.493274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.493241 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9thx\" (UniqueName: \"kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-kube-api-access-f9thx\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:52.493274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.493273 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:52.493274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.493283 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:52.493512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.493291 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:52.493512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:52.493299 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:53.211741 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:53.211707 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" Oct 30 17:33:53.212211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:53.211747 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4" event={"ID":"6c9edd23-7c40-4ee3-aa06-d68cc6800ab9","Type":"ContainerDied","Data":"bb4aba34656ef35b2882e83dec178921517994f0beaf53c19df6086f40e7ee38"} Oct 30 17:33:53.212211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:53.211813 4828 scope.go:122] "RemoveContainer" containerID="370b6afdec1d5c13505e239690ae5899fb4404b6c66d1fb5a0b48957ca7c4eb9" Oct 30 17:33:53.287402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:53.287362 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4"] Oct 30 17:33:53.318245 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:53.318211 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-41cb25-openshift-default-56c6c498b9-fv9g4"] Oct 30 17:33:54.121923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:54.108284 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:54.492775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:54.492698 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" path="/var/lib/kubelet/pods/6c9edd23-7c40-4ee3-aa06-d68cc6800ab9/volumes" Oct 30 17:33:55.127520 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.127487 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:33:55.136842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.136813 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-kube-api-access-8k2gs\" (UniqueName: \"kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-kube-api-access-8k2gs\") pod \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " Oct 30 17:33:55.137061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.136915 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-certs\") pod \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " Oct 30 17:33:55.137061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.136945 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-token\" (UniqueName: \"kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-token\") pod \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " Oct 30 17:33:55.137061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.136993 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-podinfo\") pod \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " Oct 30 17:33:55.137061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.137029 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-envoy\") pod \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " Oct 30 17:33:55.137290 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.137153 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-data\") pod \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " Oct 30 17:33:55.137290 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.137198 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-socket\") pod \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " Oct 30 17:33:55.137290 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.137273 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istiod-ca-cert\") pod \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " Oct 30 17:33:55.137447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.137395 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-credential-socket\") pod \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\" (UID: \"f3f06243-2a82-42f6-aeaa-a187b71b4dab\") " Oct 30 17:33:55.138189 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.138162 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-data" pod "f3f06243-2a82-42f6-aeaa-a187b71b4dab" (UID: "f3f06243-2a82-42f6-aeaa-a187b71b4dab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:55.138330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.138152 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-certs" pod "f3f06243-2a82-42f6-aeaa-a187b71b4dab" (UID: "f3f06243-2a82-42f6-aeaa-a187b71b4dab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:55.138330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.138245 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-credential-socket" pod "f3f06243-2a82-42f6-aeaa-a187b71b4dab" (UID: "f3f06243-2a82-42f6-aeaa-a187b71b4dab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:55.138758 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.138718 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istiod-ca-cert" pod "f3f06243-2a82-42f6-aeaa-a187b71b4dab" (UID: "f3f06243-2a82-42f6-aeaa-a187b71b4dab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:33:55.138965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.138938 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-socket" pod "f3f06243-2a82-42f6-aeaa-a187b71b4dab" (UID: "f3f06243-2a82-42f6-aeaa-a187b71b4dab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:55.139581 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.139552 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-kube-api-access-8k2gs" pod "f3f06243-2a82-42f6-aeaa-a187b71b4dab" (UID: "f3f06243-2a82-42f6-aeaa-a187b71b4dab"). InnerVolumeSpecName "kube-api-access-8k2gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:55.139703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.139648 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-podinfo" pod "f3f06243-2a82-42f6-aeaa-a187b71b4dab" (UID: "f3f06243-2a82-42f6-aeaa-a187b71b4dab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:33:55.140068 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.140050 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-envoy" pod "f3f06243-2a82-42f6-aeaa-a187b71b4dab" (UID: "f3f06243-2a82-42f6-aeaa-a187b71b4dab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:33:55.140377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.140355 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-token" pod "f3f06243-2a82-42f6-aeaa-a187b71b4dab" (UID: "f3f06243-2a82-42f6-aeaa-a187b71b4dab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:33:55.238372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.238335 4828 generic.go:356] "Generic (PLEG): container finished" podID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerID="072c9e8f1dcdee248f6e30898e5cf376293bc831b809f366e0a0b3843830a76a" exitCode=0 Oct 30 17:33:55.238578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.238401 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" Oct 30 17:33:55.238578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.238426 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" event={"ID":"f3f06243-2a82-42f6-aeaa-a187b71b4dab","Type":"ContainerDied","Data":"072c9e8f1dcdee248f6e30898e5cf376293bc831b809f366e0a0b3843830a76a"} Oct 30 17:33:55.238578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.238475 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d" event={"ID":"f3f06243-2a82-42f6-aeaa-a187b71b4dab","Type":"ContainerDied","Data":"6c4bcab1101ff58ff96af98b4c35ca76f7c4a3eee57389045d67f0bd31e8a051"} Oct 30 17:33:55.238578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.238503 4828 scope.go:122] "RemoveContainer" containerID="072c9e8f1dcdee248f6e30898e5cf376293bc831b809f366e0a0b3843830a76a" Oct 30 17:33:55.256098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.252505 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:55.256098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.252539 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:55.256098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.253586 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k2gs\" (UniqueName: \"kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-kube-api-access-8k2gs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:55.256098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.253724 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:55.256098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.253748 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:55.256098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.253778 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:55.256098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.253799 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:55.256098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.253814 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:55.256098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.253827 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f06243-2a82-42f6-aeaa-a187b71b4dab-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:33:55.275953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.275878 4828 scope.go:122] "RemoveContainer" containerID="1f94ee67b7770ee0a8c0c2c951a5f0942a9e18b4e64dad5ee154f2d34dd144f0" Oct 30 17:33:55.287667 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.287639 4828 scope.go:122] "RemoveContainer" containerID="072c9e8f1dcdee248f6e30898e5cf376293bc831b809f366e0a0b3843830a76a" Oct 30 17:33:55.288008 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:33:55.287985 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"072c9e8f1dcdee248f6e30898e5cf376293bc831b809f366e0a0b3843830a76a\": container with ID starting with 072c9e8f1dcdee248f6e30898e5cf376293bc831b809f366e0a0b3843830a76a not found: ID does not exist" containerID="072c9e8f1dcdee248f6e30898e5cf376293bc831b809f366e0a0b3843830a76a" Oct 30 17:33:55.288086 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.288019 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"072c9e8f1dcdee248f6e30898e5cf376293bc831b809f366e0a0b3843830a76a"} err="failed to get container status \"072c9e8f1dcdee248f6e30898e5cf376293bc831b809f366e0a0b3843830a76a\": rpc error: code = NotFound desc = could not find container \"072c9e8f1dcdee248f6e30898e5cf376293bc831b809f366e0a0b3843830a76a\": container with ID starting with 072c9e8f1dcdee248f6e30898e5cf376293bc831b809f366e0a0b3843830a76a not found: ID does not exist" Oct 30 17:33:55.288086 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.288038 4828 scope.go:122] "RemoveContainer" containerID="1f94ee67b7770ee0a8c0c2c951a5f0942a9e18b4e64dad5ee154f2d34dd144f0" Oct 30 17:33:55.288311 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:33:55.288279 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f94ee67b7770ee0a8c0c2c951a5f0942a9e18b4e64dad5ee154f2d34dd144f0\": container with ID starting with 1f94ee67b7770ee0a8c0c2c951a5f0942a9e18b4e64dad5ee154f2d34dd144f0 not found: ID does not exist" containerID="1f94ee67b7770ee0a8c0c2c951a5f0942a9e18b4e64dad5ee154f2d34dd144f0" Oct 30 17:33:55.288352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.288323 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f94ee67b7770ee0a8c0c2c951a5f0942a9e18b4e64dad5ee154f2d34dd144f0"} err="failed to get container status \"1f94ee67b7770ee0a8c0c2c951a5f0942a9e18b4e64dad5ee154f2d34dd144f0\": rpc error: code = NotFound desc = could not find container \"1f94ee67b7770ee0a8c0c2c951a5f0942a9e18b4e64dad5ee154f2d34dd144f0\": container with ID starting with 1f94ee67b7770ee0a8c0c2c951a5f0942a9e18b4e64dad5ee154f2d34dd144f0 not found: ID does not exist" Oct 30 17:33:55.304638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.304609 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d"] Oct 30 17:33:55.309216 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:55.309191 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d2d985-openshift-default-b6d96c8f9-dnp2d"] Oct 30 17:33:56.495770 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:56.495730 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" path="/var/lib/kubelet/pods/f3f06243-2a82-42f6-aeaa-a187b71b4dab/volumes" Oct 30 17:33:56.524564 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:56.524521 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:56.684946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:56.684845 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:58.473931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:58.470646 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:33:59.879820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:33:59.879784 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:00.174245 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:00.174151 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:00.556083 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:00.556045 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:02.167739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:02.150965 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:04.210967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:04.210738 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:06.168751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.168705 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6"] Oct 30 17:34:06.193213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.193158 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" podUID="f11b518b-9d3f-4422-98a3-82b684e2881e" containerName="maas-api" containerID="cri-o://d3c0dc41aa8d513c63c93e876584d25e92d6992d5d30e8fa498f723920532047" gracePeriod=30 Oct 30 17:34:06.272135 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.272096 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:06.395156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.394810 4828 generic.go:356] "Generic (PLEG): container finished" podID="f11b518b-9d3f-4422-98a3-82b684e2881e" containerID="d3c0dc41aa8d513c63c93e876584d25e92d6992d5d30e8fa498f723920532047" exitCode=0 Oct 30 17:34:06.395156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.395059 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" event={"ID":"f11b518b-9d3f-4422-98a3-82b684e2881e","Type":"ContainerDied","Data":"d3c0dc41aa8d513c63c93e876584d25e92d6992d5d30e8fa498f723920532047"} Oct 30 17:34:06.446827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.446786 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:06.586785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.586754 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:34:06.619717 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.615344 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:06.928956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.928877 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk"] Oct 30 17:34:06.941928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.941787 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f11b518b-9d3f-4422-98a3-82b684e2881e-kube-api-access-bp2n8\" (UniqueName: \"kubernetes.io/projected/f11b518b-9d3f-4422-98a3-82b684e2881e-kube-api-access-bp2n8\") pod \"f11b518b-9d3f-4422-98a3-82b684e2881e\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " Oct 30 17:34:06.941928 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.941921 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-maas-api-tls\") pod \"f11b518b-9d3f-4422-98a3-82b684e2881e\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " Oct 30 17:34:06.942177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.941997 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-metrics-certs\") pod \"f11b518b-9d3f-4422-98a3-82b684e2881e\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " Oct 30 17:34:06.942177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.942102 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f11b518b-9d3f-4422-98a3-82b684e2881e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11b518b-9d3f-4422-98a3-82b684e2881e-openshift-service-ca\") pod \"f11b518b-9d3f-4422-98a3-82b684e2881e\" (UID: \"f11b518b-9d3f-4422-98a3-82b684e2881e\") " Oct 30 17:34:06.943134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.943089 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f11b518b-9d3f-4422-98a3-82b684e2881e-openshift-service-ca" pod "f11b518b-9d3f-4422-98a3-82b684e2881e" (UID: "f11b518b-9d3f-4422-98a3-82b684e2881e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:06.947139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.946968 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-metrics-certs" pod "f11b518b-9d3f-4422-98a3-82b684e2881e" (UID: "f11b518b-9d3f-4422-98a3-82b684e2881e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:06.947139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.946999 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-maas-api-tls" pod "f11b518b-9d3f-4422-98a3-82b684e2881e" (UID: "f11b518b-9d3f-4422-98a3-82b684e2881e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:06.947139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:06.947100 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f11b518b-9d3f-4422-98a3-82b684e2881e-kube-api-access-bp2n8" pod "f11b518b-9d3f-4422-98a3-82b684e2881e" (UID: "f11b518b-9d3f-4422-98a3-82b684e2881e"). InnerVolumeSpecName "kube-api-access-bp2n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:07.004792 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.004643 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:07.044597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.044552 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp2n8\" (UniqueName: \"kubernetes.io/projected/f11b518b-9d3f-4422-98a3-82b684e2881e-kube-api-access-bp2n8\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:07.044597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.044597 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:07.045048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.044610 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f11b518b-9d3f-4422-98a3-82b684e2881e-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:07.045048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.044621 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f11b518b-9d3f-4422-98a3-82b684e2881e-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:07.173221 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.172998 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:07.369091 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.369050 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:07.407089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.407030 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" event={"ID":"f11b518b-9d3f-4422-98a3-82b684e2881e","Type":"ContainerDied","Data":"f4d1513eb2525c0b77d76684868992922eccdd4db14b2d6adaa8d9815f1b6114"} Oct 30 17:34:07.407310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.407109 4828 scope.go:122] "RemoveContainer" containerID="d3c0dc41aa8d513c63c93e876584d25e92d6992d5d30e8fa498f723920532047" Oct 30 17:34:07.407310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.407288 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6" Oct 30 17:34:07.466276 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.466239 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6"] Oct 30 17:34:07.471507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.471470 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-93e020-7df85fd9df-c8gd6"] Oct 30 17:34:07.524751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.524707 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:07.847465 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:07.847428 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:07.976196 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:34:07.976156 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:34:08.494623 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:08.494592 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f11b518b-9d3f-4422-98a3-82b684e2881e" path="/var/lib/kubelet/pods/f11b518b-9d3f-4422-98a3-82b684e2881e/volumes" Oct 30 17:34:08.661925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:08.661217 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:08.825380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:08.825335 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:09.404921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:09.390356 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:09.832521 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:09.832486 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:11.118334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:11.112167 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:11.930394 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:11.930334 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" podUID="727b046f-b952-46d6-ad72-c66eae2dac0b" containerName="payload-pre-processing" containerID="cri-o://eadc6758d09874305a9c9f36968f6021d40ab258faea2c41fe05e4c4089db59b" gracePeriod=25 Oct 30 17:34:12.251395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.251362 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" Oct 30 17:34:12.271396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.271359 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727b046f-b952-46d6-ad72-c66eae2dac0b-kube-api-access-8nlxj\" (UniqueName: \"kubernetes.io/projected/727b046f-b952-46d6-ad72-c66eae2dac0b-kube-api-access-8nlxj\") pod \"727b046f-b952-46d6-ad72-c66eae2dac0b\" (UID: \"727b046f-b952-46d6-ad72-c66eae2dac0b\") " Oct 30 17:34:12.271606 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.271428 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/727b046f-b952-46d6-ad72-c66eae2dac0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b046f-b952-46d6-ad72-c66eae2dac0b-plugins-config-volume\") pod \"727b046f-b952-46d6-ad72-c66eae2dac0b\" (UID: \"727b046f-b952-46d6-ad72-c66eae2dac0b\") " Oct 30 17:34:12.272280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.272168 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727b046f-b952-46d6-ad72-c66eae2dac0b-plugins-config-volume" pod "727b046f-b952-46d6-ad72-c66eae2dac0b" (UID: "727b046f-b952-46d6-ad72-c66eae2dac0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:12.272280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.272257 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727b046f-b952-46d6-ad72-c66eae2dac0b-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:12.274269 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.274233 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727b046f-b952-46d6-ad72-c66eae2dac0b-kube-api-access-8nlxj" pod "727b046f-b952-46d6-ad72-c66eae2dac0b" (UID: "727b046f-b952-46d6-ad72-c66eae2dac0b"). InnerVolumeSpecName "kube-api-access-8nlxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:12.374215 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.374172 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nlxj\" (UniqueName: \"kubernetes.io/projected/727b046f-b952-46d6-ad72-c66eae2dac0b-kube-api-access-8nlxj\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:12.505462 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.505368 4828 generic.go:356] "Generic (PLEG): container finished" podID="727b046f-b952-46d6-ad72-c66eae2dac0b" containerID="eadc6758d09874305a9c9f36968f6021d40ab258faea2c41fe05e4c4089db59b" exitCode=0 Oct 30 17:34:12.505462 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.505417 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" Oct 30 17:34:12.505669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.505407 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" event={"ID":"727b046f-b952-46d6-ad72-c66eae2dac0b","Type":"ContainerDied","Data":"eadc6758d09874305a9c9f36968f6021d40ab258faea2c41fe05e4c4089db59b"} Oct 30 17:34:12.505669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.505563 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk" event={"ID":"727b046f-b952-46d6-ad72-c66eae2dac0b","Type":"ContainerDied","Data":"0262fad146d71a1a553bb8b12bbb610e6c820c527c400de54548cb2cc00677f1"} Oct 30 17:34:12.505669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.505595 4828 scope.go:122] "RemoveContainer" containerID="eadc6758d09874305a9c9f36968f6021d40ab258faea2c41fe05e4c4089db59b" Oct 30 17:34:12.520021 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.519995 4828 scope.go:122] "RemoveContainer" containerID="eadc6758d09874305a9c9f36968f6021d40ab258faea2c41fe05e4c4089db59b" Oct 30 17:34:12.520313 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:34:12.520293 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eadc6758d09874305a9c9f36968f6021d40ab258faea2c41fe05e4c4089db59b\": container with ID starting with eadc6758d09874305a9c9f36968f6021d40ab258faea2c41fe05e4c4089db59b not found: ID does not exist" containerID="eadc6758d09874305a9c9f36968f6021d40ab258faea2c41fe05e4c4089db59b" Oct 30 17:34:12.520397 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.520320 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eadc6758d09874305a9c9f36968f6021d40ab258faea2c41fe05e4c4089db59b"} err="failed to get container status \"eadc6758d09874305a9c9f36968f6021d40ab258faea2c41fe05e4c4089db59b\": rpc error: code = NotFound desc = could not find container \"eadc6758d09874305a9c9f36968f6021d40ab258faea2c41fe05e4c4089db59b\": container with ID starting with eadc6758d09874305a9c9f36968f6021d40ab258faea2c41fe05e4c4089db59b not found: ID does not exist" Oct 30 17:34:12.562703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.562680 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk"] Oct 30 17:34:12.575419 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.575387 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-5b5b57648c-x7rxk"] Oct 30 17:34:12.978110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:12.978073 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:13.629625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:13.623270 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:14.303210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.303120 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:14.496433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.496393 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="727b046f-b952-46d6-ad72-c66eae2dac0b" path="/var/lib/kubelet/pods/727b046f-b952-46d6-ad72-c66eae2dac0b/volumes" Oct 30 17:34:14.563458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.563381 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.809267 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf"] Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811069 4828 cpu_manager.go:414] "containerMap: removing container" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811094 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811108 4828 cpu_manager.go:414] "containerMap: removing container" podUID="e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811116 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811139 4828 cpu_manager.go:414] "containerMap: removing container" podUID="785fe8ec-352e-4c68-ad24-924106f4dc53" containerName="payload-processing" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811148 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="785fe8ec-352e-4c68-ad24-924106f4dc53" containerName="payload-processing" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811169 4828 cpu_manager.go:414] "containerMap: removing container" podUID="f11b518b-9d3f-4422-98a3-82b684e2881e" containerName="maas-api" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811177 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f11b518b-9d3f-4422-98a3-82b684e2881e" containerName="maas-api" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811193 4828 cpu_manager.go:414] "containerMap: removing container" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811198 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811206 4828 cpu_manager.go:414] "containerMap: removing container" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811210 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811219 4828 cpu_manager.go:414] "containerMap: removing container" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811228 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811240 4828 cpu_manager.go:414] "containerMap: removing container" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811247 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811257 4828 cpu_manager.go:414] "containerMap: removing container" podUID="694fc7b7-12ff-4f40-a513-44fd85f92439" containerName="maas-api" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811262 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="694fc7b7-12ff-4f40-a513-44fd85f92439" containerName="maas-api" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811294 4828 cpu_manager.go:414] "containerMap: removing container" podUID="fac5920b-1b21-4328-ae7f-4586183f34c7" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811298 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac5920b-1b21-4328-ae7f-4586183f34c7" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811305 4828 cpu_manager.go:414] "containerMap: removing container" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811309 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811324 4828 cpu_manager.go:414] "containerMap: removing container" podUID="727b046f-b952-46d6-ad72-c66eae2dac0b" containerName="payload-pre-processing" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811332 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727b046f-b952-46d6-ad72-c66eae2dac0b" containerName="payload-pre-processing" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811484 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="f11b518b-9d3f-4422-98a3-82b684e2881e" containerName="maas-api" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811498 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811504 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9edd23-7c40-4ee3-aa06-d68cc6800ab9" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811511 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811518 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f58c09-e14a-41e4-a355-c08c7575c648" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811524 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="694fc7b7-12ff-4f40-a513-44fd85f92439" containerName="maas-api" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811535 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac5920b-1b21-4328-ae7f-4586183f34c7" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811541 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811546 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="727b046f-b952-46d6-ad72-c66eae2dac0b" containerName="payload-pre-processing" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811561 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49de0e3-8b4f-4f8a-b14c-aa94892f6dee" containerName="istio-proxy" Oct 30 17:34:14.814548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.811574 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="785fe8ec-352e-4c68-ad24-924106f4dc53" containerName="payload-processing" Oct 30 17:34:14.823590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.823562 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:14.842370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.842302 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-8f92ef-openshift-default-dockercfg-g4zmc\"" Oct 30 17:34:14.846160 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.845470 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf"] Oct 30 17:34:14.890769 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.890727 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:14.983927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.981997 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istiod-ca-cert\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:14.983927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.982115 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-certs\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:14.983927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.982227 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-podinfo\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:14.983927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.982263 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-data\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:14.983927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.982329 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-socket\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:14.983927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.982457 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-token\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:14.983927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.982549 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf6bb\" (UniqueName: \"kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-kube-api-access-vf6bb\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:14.983927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.982635 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-credential-socket\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:14.983927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:14.982670 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-envoy\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.080047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.067286 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:15.087920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.087034 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-certs\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.087920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.087144 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-podinfo\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.087920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.087178 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-data\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.087920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.087232 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-socket\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.087920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.087337 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-token\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.087920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.087415 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf6bb\" (UniqueName: \"kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-kube-api-access-vf6bb\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.087920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.087484 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-credential-socket\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.087920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.087511 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-envoy\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.087920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.087684 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istiod-ca-cert\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.088627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.088587 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istiod-ca-cert\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.115925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.088881 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-certs\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.115925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.107097 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-data\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.115925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.107356 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-socket\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.115925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.107657 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-credential-socket\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.115925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.115472 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-envoy\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.135436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.135293 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-podinfo\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.142954 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.140425 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf6bb\" (UniqueName: \"kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-kube-api-access-vf6bb\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.156932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.150006 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-token\") pod \"e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.171001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.166949 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:15.221807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.221770 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:15.395857 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.395536 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:15.631738 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.631604 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf"] Oct 30 17:34:15.654364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.654315 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:34:15.654544 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.654466 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:34:15.654631 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.654601 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:34:15.667403 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.667377 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672"] Oct 30 17:34:15.667687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.667663 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" containerID="cri-o://7f0b75f97d03b9c3833c8e05c4553ac6106a4f61d57a9af0099ff0f1ee0f11f0" gracePeriod=30 Oct 30 17:34:15.714703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:15.714646 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:16.544372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:16.544328 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" event={"ID":"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5","Type":"ContainerStarted","Data":"fa94b91391edcffb19b35439803dad5a28e1676757939c06affb8413468de3a8"} Oct 30 17:34:16.544789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:16.544395 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" event={"ID":"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5","Type":"ContainerStarted","Data":"056dc9681adc5fb3e738225b3a78859d36b46bbef362bcf280b437ebb95de278"} Oct 30 17:34:16.556948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:16.556868 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:16.607191 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:16.607121 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" podStartSLOduration=2.607098427 podStartE2EDuration="2.607098427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:16.602682949 +0000 UTC m=+2834.248193130" watchObservedRunningTime="2026-10-30 17:34:16.607098427 +0000 UTC m=+2834.252608607" Oct 30 17:34:16.746991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:16.746951 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:16.930937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:16.930794 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:34:16.931108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:16.930879 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:34:17.168558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:17.168523 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:17.175363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:17.175331 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:17.554033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:17.553996 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:17.554969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:17.554943 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:34:18.724264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:18.719415 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:19.137949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:19.135382 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:19.303572 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:19.303535 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:20.812086 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:20.812030 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:20.994791 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:20.994752 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:21.045390 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.045337 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c"] Oct 30 17:34:21.046952 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.046925 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f06243-2a82-42f6-aeaa-a187b71b4dab" containerName="istio-proxy" Oct 30 17:34:21.052007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.051982 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" Oct 30 17:34:21.054732 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.054704 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c\" (UID: \"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" Oct 30 17:34:21.054856 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.054835 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp8wr\" (UniqueName: \"kubernetes.io/projected/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-kube-api-access-lp8wr\") pod \"payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c\" (UID: \"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" Oct 30 17:34:21.078331 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.078294 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c"] Oct 30 17:34:21.083730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.082609 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8f92ef\"" Oct 30 17:34:21.083730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.082712 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8f92ef-dockercfg-gxm6b\"" Oct 30 17:34:21.088636 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.088605 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:34:21.189925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.188988 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-socket\") pod \"914537dc-831c-4b47-99fe-ce17f41b6247\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " Oct 30 17:34:21.189925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.189163 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-envoy\") pod \"914537dc-831c-4b47-99fe-ce17f41b6247\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " Oct 30 17:34:21.189925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.189223 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-certs\") pod \"914537dc-831c-4b47-99fe-ce17f41b6247\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " Oct 30 17:34:21.189925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.189259 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-credential-socket\") pod \"914537dc-831c-4b47-99fe-ce17f41b6247\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " Oct 30 17:34:21.189925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.189320 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-istio-token\" (UniqueName: \"kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-istio-token\") pod \"914537dc-831c-4b47-99fe-ce17f41b6247\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " Oct 30 17:34:21.189925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.189401 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-kube-api-access-2qkp9\" (UniqueName: \"kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-kube-api-access-2qkp9\") pod \"914537dc-831c-4b47-99fe-ce17f41b6247\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " Oct 30 17:34:21.189925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.189577 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914537dc-831c-4b47-99fe-ce17f41b6247-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914537dc-831c-4b47-99fe-ce17f41b6247-istiod-ca-cert\") pod \"914537dc-831c-4b47-99fe-ce17f41b6247\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " Oct 30 17:34:21.189925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.189662 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/914537dc-831c-4b47-99fe-ce17f41b6247-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914537dc-831c-4b47-99fe-ce17f41b6247-istio-podinfo\") pod \"914537dc-831c-4b47-99fe-ce17f41b6247\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " Oct 30 17:34:21.189925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.189693 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-data\") pod \"914537dc-831c-4b47-99fe-ce17f41b6247\" (UID: \"914537dc-831c-4b47-99fe-ce17f41b6247\") " Oct 30 17:34:21.190518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.190035 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c\" (UID: \"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" Oct 30 17:34:21.190518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.190240 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp8wr\" (UniqueName: \"kubernetes.io/projected/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-kube-api-access-lp8wr\") pod \"payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c\" (UID: \"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" Oct 30 17:34:21.197929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.196986 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-socket" pod "914537dc-831c-4b47-99fe-ce17f41b6247" (UID: "914537dc-831c-4b47-99fe-ce17f41b6247"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:21.258299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.255461 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914537dc-831c-4b47-99fe-ce17f41b6247-istiod-ca-cert" pod "914537dc-831c-4b47-99fe-ce17f41b6247" (UID: "914537dc-831c-4b47-99fe-ce17f41b6247"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:21.342004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.334840 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq"] Oct 30 17:34:21.342004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.336186 4828 cpu_manager.go:414] "containerMap: removing container" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" Oct 30 17:34:21.342004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.336211 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" Oct 30 17:34:21.342004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.336409 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" Oct 30 17:34:21.342004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.336431 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" Oct 30 17:34:21.367936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.352765 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-data" pod "914537dc-831c-4b47-99fe-ce17f41b6247" (UID: "914537dc-831c-4b47-99fe-ce17f41b6247"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:21.367936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.353700 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c\" (UID: \"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" Oct 30 17:34:21.367936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.355329 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:21.367936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.355355 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:21.367936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.355373 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914537dc-831c-4b47-99fe-ce17f41b6247-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:21.367936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.356435 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-credential-socket" pod "914537dc-831c-4b47-99fe-ce17f41b6247" (UID: "914537dc-831c-4b47-99fe-ce17f41b6247"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:21.367936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.356616 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-certs" pod "914537dc-831c-4b47-99fe-ce17f41b6247" (UID: "914537dc-831c-4b47-99fe-ce17f41b6247"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:21.381405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.379047 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-envoy" pod "914537dc-831c-4b47-99fe-ce17f41b6247" (UID: "914537dc-831c-4b47-99fe-ce17f41b6247"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:21.381405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.380585 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-kube-api-access-2qkp9" pod "914537dc-831c-4b47-99fe-ce17f41b6247" (UID: "914537dc-831c-4b47-99fe-ce17f41b6247"). InnerVolumeSpecName "kube-api-access-2qkp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:21.381405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.380656 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp8wr\" (UniqueName: \"kubernetes.io/projected/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-kube-api-access-lp8wr\") pod \"payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c\" (UID: \"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" Oct 30 17:34:21.381766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.381695 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/914537dc-831c-4b47-99fe-ce17f41b6247-istio-podinfo" pod "914537dc-831c-4b47-99fe-ce17f41b6247" (UID: "914537dc-831c-4b47-99fe-ce17f41b6247"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:34:21.384641 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.383205 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" Oct 30 17:34:21.384818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.383089 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq"] Oct 30 17:34:21.394876 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.391647 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-istio-token" pod "914537dc-831c-4b47-99fe-ce17f41b6247" (UID: "914537dc-831c-4b47-99fe-ce17f41b6247"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:21.405485 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.405454 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" Oct 30 17:34:21.456890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.456858 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:21.456890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.456887 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:21.456890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.456911 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914537dc-831c-4b47-99fe-ce17f41b6247-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:21.456890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.456926 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:21.457202 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.456937 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qkp9\" (UniqueName: \"kubernetes.io/projected/914537dc-831c-4b47-99fe-ce17f41b6247-kube-api-access-2qkp9\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:21.457202 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.456946 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914537dc-831c-4b47-99fe-ce17f41b6247-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:21.560391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.557959 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a43570-25ea-4738-9109-85aaf132547d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq\" (UID: \"32a43570-25ea-4738-9109-85aaf132547d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" Oct 30 17:34:21.560391 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.558181 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2cg2\" (UniqueName: \"kubernetes.io/projected/32a43570-25ea-4738-9109-85aaf132547d-kube-api-access-b2cg2\") pod \"payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq\" (UID: \"32a43570-25ea-4738-9109-85aaf132547d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" Oct 30 17:34:21.594340 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.594302 4828 generic.go:356] "Generic (PLEG): container finished" podID="914537dc-831c-4b47-99fe-ce17f41b6247" containerID="7f0b75f97d03b9c3833c8e05c4553ac6106a4f61d57a9af0099ff0f1ee0f11f0" exitCode=0 Oct 30 17:34:21.594549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.594371 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" Oct 30 17:34:21.594549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.594379 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" event={"ID":"914537dc-831c-4b47-99fe-ce17f41b6247","Type":"ContainerDied","Data":"7f0b75f97d03b9c3833c8e05c4553ac6106a4f61d57a9af0099ff0f1ee0f11f0"} Oct 30 17:34:21.594549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.594427 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672" event={"ID":"914537dc-831c-4b47-99fe-ce17f41b6247","Type":"ContainerDied","Data":"ea177efb2ea10e455f1328ad45a57740f98e0c51a666cc6f50e72323f3cf6c0d"} Oct 30 17:34:21.594549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.594451 4828 scope.go:122] "RemoveContainer" containerID="7f0b75f97d03b9c3833c8e05c4553ac6106a4f61d57a9af0099ff0f1ee0f11f0" Oct 30 17:34:21.608943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.608876 4828 scope.go:122] "RemoveContainer" containerID="a2556c4456fbb593285cfa3478bd4921e709399b03d161cbbb0580315e3f446f" Oct 30 17:34:21.623594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.623570 4828 scope.go:122] "RemoveContainer" containerID="7f0b75f97d03b9c3833c8e05c4553ac6106a4f61d57a9af0099ff0f1ee0f11f0" Oct 30 17:34:21.623976 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:34:21.623953 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f0b75f97d03b9c3833c8e05c4553ac6106a4f61d57a9af0099ff0f1ee0f11f0\": container with ID starting with 7f0b75f97d03b9c3833c8e05c4553ac6106a4f61d57a9af0099ff0f1ee0f11f0 not found: ID does not exist" containerID="7f0b75f97d03b9c3833c8e05c4553ac6106a4f61d57a9af0099ff0f1ee0f11f0" Oct 30 17:34:21.624052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.623983 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f0b75f97d03b9c3833c8e05c4553ac6106a4f61d57a9af0099ff0f1ee0f11f0"} err="failed to get container status \"7f0b75f97d03b9c3833c8e05c4553ac6106a4f61d57a9af0099ff0f1ee0f11f0\": rpc error: code = NotFound desc = could not find container \"7f0b75f97d03b9c3833c8e05c4553ac6106a4f61d57a9af0099ff0f1ee0f11f0\": container with ID starting with 7f0b75f97d03b9c3833c8e05c4553ac6106a4f61d57a9af0099ff0f1ee0f11f0 not found: ID does not exist" Oct 30 17:34:21.624052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.624002 4828 scope.go:122] "RemoveContainer" containerID="a2556c4456fbb593285cfa3478bd4921e709399b03d161cbbb0580315e3f446f" Oct 30 17:34:21.624326 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:34:21.624296 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2556c4456fbb593285cfa3478bd4921e709399b03d161cbbb0580315e3f446f\": container with ID starting with a2556c4456fbb593285cfa3478bd4921e709399b03d161cbbb0580315e3f446f not found: ID does not exist" containerID="a2556c4456fbb593285cfa3478bd4921e709399b03d161cbbb0580315e3f446f" Oct 30 17:34:21.624378 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.624330 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2556c4456fbb593285cfa3478bd4921e709399b03d161cbbb0580315e3f446f"} err="failed to get container status \"a2556c4456fbb593285cfa3478bd4921e709399b03d161cbbb0580315e3f446f\": rpc error: code = NotFound desc = could not find container \"a2556c4456fbb593285cfa3478bd4921e709399b03d161cbbb0580315e3f446f\": container with ID starting with a2556c4456fbb593285cfa3478bd4921e709399b03d161cbbb0580315e3f446f not found: ID does not exist" Oct 30 17:34:21.659409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.659373 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2cg2\" (UniqueName: \"kubernetes.io/projected/32a43570-25ea-4738-9109-85aaf132547d-kube-api-access-b2cg2\") pod \"payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq\" (UID: \"32a43570-25ea-4738-9109-85aaf132547d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" Oct 30 17:34:21.659642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.659549 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a43570-25ea-4738-9109-85aaf132547d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq\" (UID: \"32a43570-25ea-4738-9109-85aaf132547d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" Oct 30 17:34:21.660331 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.660304 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a43570-25ea-4738-9109-85aaf132547d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq\" (UID: \"32a43570-25ea-4738-9109-85aaf132547d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" Oct 30 17:34:21.667043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.667016 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672"] Oct 30 17:34:21.672267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.672229 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-93e020-openshift-default-6f85fc494c-s4672"] Oct 30 17:34:21.689973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.689937 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2cg2\" (UniqueName: \"kubernetes.io/projected/32a43570-25ea-4738-9109-85aaf132547d-kube-api-access-b2cg2\") pod \"payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq\" (UID: \"32a43570-25ea-4738-9109-85aaf132547d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" Oct 30 17:34:21.715634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.715595 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" Oct 30 17:34:21.740382 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:34:21.740347 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84d9cf0_b5f6_4e24_b189_4e80e341e9f9.slice/crio-5c5c3aac454fa2c0295ec285751a6d47d567e9b1b82de8b02a45cb3c4de785fd WatchSource:0}: Error finding container 5c5c3aac454fa2c0295ec285751a6d47d567e9b1b82de8b02a45cb3c4de785fd: Status 404 returned error can't find the container with id 5c5c3aac454fa2c0295ec285751a6d47d567e9b1b82de8b02a45cb3c4de785fd Oct 30 17:34:21.741371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:21.741344 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c"] Oct 30 17:34:22.035262 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:34:22.035232 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a43570_25ea_4738_9109_85aaf132547d.slice/crio-613739da28811c633cfcd4f758d2b65d6673755fe6c277d1942a5668c34ab937 WatchSource:0}: Error finding container 613739da28811c633cfcd4f758d2b65d6673755fe6c277d1942a5668c34ab937: Status 404 returned error can't find the container with id 613739da28811c633cfcd4f758d2b65d6673755fe6c277d1942a5668c34ab937 Oct 30 17:34:22.037365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.037336 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq"] Oct 30 17:34:22.339772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.339732 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:22.518941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.517505 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:22.540359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.540325 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" path="/var/lib/kubelet/pods/914537dc-831c-4b47-99fe-ce17f41b6247/volumes" Oct 30 17:34:22.617387 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.614785 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" event={"ID":"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9","Type":"ContainerStarted","Data":"0c4e042fd4fc2d4648f994adac711aff10142290b4b6b10a322f6e3fd9234bb7"} Oct 30 17:34:22.617387 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.614872 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" event={"ID":"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9","Type":"ContainerStarted","Data":"5c5c3aac454fa2c0295ec285751a6d47d567e9b1b82de8b02a45cb3c4de785fd"} Oct 30 17:34:22.617387 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.615198 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" Oct 30 17:34:22.618785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.618031 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" event={"ID":"32a43570-25ea-4738-9109-85aaf132547d","Type":"ContainerStarted","Data":"493640af5ea80450e1df546ec795e81378edbccc2c6cb0a4d049fc3b496be0b8"} Oct 30 17:34:22.618785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.618098 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" event={"ID":"32a43570-25ea-4738-9109-85aaf132547d","Type":"ContainerStarted","Data":"613739da28811c633cfcd4f758d2b65d6673755fe6c277d1942a5668c34ab937"} Oct 30 17:34:22.619184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.619161 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" Oct 30 17:34:22.768640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.768576 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" podStartSLOduration=1.768557055 podStartE2EDuration="1.768557055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:22.685224153 +0000 UTC m=+2840.330734333" watchObservedRunningTime="2026-10-30 17:34:22.768557055 +0000 UTC m=+2840.414067236" Oct 30 17:34:22.772028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.771952 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb"] Oct 30 17:34:22.773438 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.773412 4828 cpu_manager.go:414] "containerMap: removing container" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" Oct 30 17:34:22.773594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.773579 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914537dc-831c-4b47-99fe-ce17f41b6247" containerName="istio-proxy" Oct 30 17:34:22.830923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.810501 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" podStartSLOduration=1.810472364 podStartE2EDuration="1.810472364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:22.786218223 +0000 UTC m=+2840.431728405" watchObservedRunningTime="2026-10-30 17:34:22.810472364 +0000 UTC m=+2840.455982571" Oct 30 17:34:22.838983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.816713 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb"] Oct 30 17:34:22.839284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.816808 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" Oct 30 17:34:22.869854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.869824 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4fb821\"" Oct 30 17:34:22.956010 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.942536 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f8f535-a840-4e43-8724-a408f694bf6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb\" (UID: \"18f8f535-a840-4e43-8724-a408f694bf6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" Oct 30 17:34:22.962977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:22.962940 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dnfn\" (UniqueName: \"kubernetes.io/projected/18f8f535-a840-4e43-8724-a408f694bf6c-kube-api-access-8dnfn\") pod \"payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb\" (UID: \"18f8f535-a840-4e43-8724-a408f694bf6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" Oct 30 17:34:23.047707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.047621 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:23.065304 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.065274 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f8f535-a840-4e43-8724-a408f694bf6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb\" (UID: \"18f8f535-a840-4e43-8724-a408f694bf6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" Oct 30 17:34:23.065473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.065339 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dnfn\" (UniqueName: \"kubernetes.io/projected/18f8f535-a840-4e43-8724-a408f694bf6c-kube-api-access-8dnfn\") pod \"payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb\" (UID: \"18f8f535-a840-4e43-8724-a408f694bf6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" Oct 30 17:34:23.066052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.066033 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f8f535-a840-4e43-8724-a408f694bf6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb\" (UID: \"18f8f535-a840-4e43-8724-a408f694bf6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" Oct 30 17:34:23.089456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.089419 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dnfn\" (UniqueName: \"kubernetes.io/projected/18f8f535-a840-4e43-8724-a408f694bf6c-kube-api-access-8dnfn\") pod \"payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb\" (UID: \"18f8f535-a840-4e43-8724-a408f694bf6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" Oct 30 17:34:23.148133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.148090 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z"] Oct 30 17:34:23.148365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.148302 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" podUID="ea2c6df0-ed33-4fff-9482-6e3cb6f6071f" containerName="maas-api" containerID="cri-o://8c58aee7b3bbeb5a05c77d233d233bf10a98fc3e49a63cc13670d5370f5ae496" gracePeriod=30 Oct 30 17:34:23.171645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.171607 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" Oct 30 17:34:23.260464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.260423 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:23.439356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.439318 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:23.516298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.516264 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:34:23.535998 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:34:23.535963 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f8f535_a840_4e43_8724_a408f694bf6c.slice/crio-cd95ee0b4e5e01225c52bc31d97137e024c107544e0f067078cb7520736d8d53 WatchSource:0}: Error finding container cd95ee0b4e5e01225c52bc31d97137e024c107544e0f067078cb7520736d8d53: Status 404 returned error can't find the container with id cd95ee0b4e5e01225c52bc31d97137e024c107544e0f067078cb7520736d8d53 Oct 30 17:34:23.540077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.540054 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb"] Oct 30 17:34:23.580946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.580861 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-maas-api-tls\") pod \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " Oct 30 17:34:23.581470 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.581435 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-metrics-certs\") pod \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " Oct 30 17:34:23.581850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.581827 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-openshift-service-ca\") pod \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " Oct 30 17:34:23.582012 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.581869 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-kube-api-access-gg8wr\" (UniqueName: \"kubernetes.io/projected/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-kube-api-access-gg8wr\") pod \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\" (UID: \"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f\") " Oct 30 17:34:23.582871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.582847 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-openshift-service-ca" pod "ea2c6df0-ed33-4fff-9482-6e3cb6f6071f" (UID: "ea2c6df0-ed33-4fff-9482-6e3cb6f6071f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:23.598023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.597986 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:23.600429 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.600395 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-maas-api-tls" pod "ea2c6df0-ed33-4fff-9482-6e3cb6f6071f" (UID: "ea2c6df0-ed33-4fff-9482-6e3cb6f6071f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:23.604377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.604341 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-metrics-certs" pod "ea2c6df0-ed33-4fff-9482-6e3cb6f6071f" (UID: "ea2c6df0-ed33-4fff-9482-6e3cb6f6071f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:23.606422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.606390 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-kube-api-access-gg8wr" pod "ea2c6df0-ed33-4fff-9482-6e3cb6f6071f" (UID: "ea2c6df0-ed33-4fff-9482-6e3cb6f6071f"). InnerVolumeSpecName "kube-api-access-gg8wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:23.633977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.633936 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" event={"ID":"18f8f535-a840-4e43-8724-a408f694bf6c","Type":"ContainerStarted","Data":"cd95ee0b4e5e01225c52bc31d97137e024c107544e0f067078cb7520736d8d53"} Oct 30 17:34:23.635779 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.635751 4828 generic.go:356] "Generic (PLEG): container finished" podID="ea2c6df0-ed33-4fff-9482-6e3cb6f6071f" containerID="8c58aee7b3bbeb5a05c77d233d233bf10a98fc3e49a63cc13670d5370f5ae496" exitCode=0 Oct 30 17:34:23.635958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.635812 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" Oct 30 17:34:23.636964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.636040 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" event={"ID":"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f","Type":"ContainerDied","Data":"8c58aee7b3bbeb5a05c77d233d233bf10a98fc3e49a63cc13670d5370f5ae496"} Oct 30 17:34:23.636964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.636090 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z" event={"ID":"ea2c6df0-ed33-4fff-9482-6e3cb6f6071f","Type":"ContainerDied","Data":"eb8206ff09100e196e62d80b8bfba2ca5f91aa1cf0a2fd079a751256e3c2dc00"} Oct 30 17:34:23.636964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.636121 4828 scope.go:122] "RemoveContainer" containerID="8c58aee7b3bbeb5a05c77d233d233bf10a98fc3e49a63cc13670d5370f5ae496" Oct 30 17:34:23.655751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.655724 4828 scope.go:122] "RemoveContainer" containerID="8c58aee7b3bbeb5a05c77d233d233bf10a98fc3e49a63cc13670d5370f5ae496" Oct 30 17:34:23.656227 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:34:23.656200 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c58aee7b3bbeb5a05c77d233d233bf10a98fc3e49a63cc13670d5370f5ae496\": container with ID starting with 8c58aee7b3bbeb5a05c77d233d233bf10a98fc3e49a63cc13670d5370f5ae496 not found: ID does not exist" containerID="8c58aee7b3bbeb5a05c77d233d233bf10a98fc3e49a63cc13670d5370f5ae496" Oct 30 17:34:23.656407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.656379 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c58aee7b3bbeb5a05c77d233d233bf10a98fc3e49a63cc13670d5370f5ae496"} err="failed to get container status \"8c58aee7b3bbeb5a05c77d233d233bf10a98fc3e49a63cc13670d5370f5ae496\": rpc error: code = NotFound desc = could not find container \"8c58aee7b3bbeb5a05c77d233d233bf10a98fc3e49a63cc13670d5370f5ae496\": container with ID starting with 8c58aee7b3bbeb5a05c77d233d233bf10a98fc3e49a63cc13670d5370f5ae496 not found: ID does not exist" Oct 30 17:34:23.694925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.689050 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:23.694925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.689087 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:23.694925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.689102 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:23.694925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.689118 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg8wr\" (UniqueName: \"kubernetes.io/projected/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f-kube-api-access-gg8wr\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:23.773668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.773600 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z"] Oct 30 17:34:23.774018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.773998 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c17e8c-f47fcbbf6-h8z7z"] Oct 30 17:34:23.905710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:23.881574 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:24.057829 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:24.057776 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:24.210880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:24.210789 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:24.495963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:24.495861 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea2c6df0-ed33-4fff-9482-6e3cb6f6071f" path="/var/lib/kubelet/pods/ea2c6df0-ed33-4fff-9482-6e3cb6f6071f/volumes" Oct 30 17:34:24.641716 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:24.641674 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:24.646784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:24.646740 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" event={"ID":"18f8f535-a840-4e43-8724-a408f694bf6c","Type":"ContainerStarted","Data":"d5cc840c1a00d18a6438c156e1cff83274d026b8d0c9ce35b398373a50d4f73f"} Oct 30 17:34:24.646972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:24.646892 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" Oct 30 17:34:24.693798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:24.693741 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" podStartSLOduration=2.693719329 podStartE2EDuration="2.693719329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:24.689612045 +0000 UTC m=+2842.335122226" watchObservedRunningTime="2026-10-30 17:34:24.693719329 +0000 UTC m=+2842.339229509" Oct 30 17:34:25.863341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:25.863299 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:26.220235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:26.220152 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8"] Oct 30 17:34:26.255647 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:26.255608 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr"] Oct 30 17:34:26.409452 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:26.409414 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:26.559934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:26.559881 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:27.624989 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:27.624949 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:29.199689 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:29.199649 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:30.989193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:30.989142 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:31.155461 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.155426 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:31.220496 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.220447 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" podUID="4eb59754-4f88-4815-bc64-b53876d50b55" containerName="payload-processing" containerID="cri-o://bcfff059c4a372fed3f88fc7c7c96a6ccac030c345f3de029b20f9ad601103bb" gracePeriod=25 Oct 30 17:34:31.256199 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.256162 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" podUID="ea7d0036-8373-47e1-98f8-70c094b0e8d9" containerName="payload-pre-processing" containerID="cri-o://6c8ed4dd4548809e57332a5260e4f03b002ec29a84d1626eaf67fe47838985bc" gracePeriod=25 Oct 30 17:34:31.322084 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.322047 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:31.724287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.723818 4828 generic.go:356] "Generic (PLEG): container finished" podID="4eb59754-4f88-4815-bc64-b53876d50b55" containerID="bcfff059c4a372fed3f88fc7c7c96a6ccac030c345f3de029b20f9ad601103bb" exitCode=0 Oct 30 17:34:31.724287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.723929 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" event={"ID":"4eb59754-4f88-4815-bc64-b53876d50b55","Type":"ContainerDied","Data":"bcfff059c4a372fed3f88fc7c7c96a6ccac030c345f3de029b20f9ad601103bb"} Oct 30 17:34:31.726105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.726078 4828 generic.go:356] "Generic (PLEG): container finished" podID="ea7d0036-8373-47e1-98f8-70c094b0e8d9" containerID="6c8ed4dd4548809e57332a5260e4f03b002ec29a84d1626eaf67fe47838985bc" exitCode=0 Oct 30 17:34:31.726246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.726145 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" event={"ID":"ea7d0036-8373-47e1-98f8-70c094b0e8d9","Type":"ContainerDied","Data":"6c8ed4dd4548809e57332a5260e4f03b002ec29a84d1626eaf67fe47838985bc"} Oct 30 17:34:31.790075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.790042 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" Oct 30 17:34:31.827950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.827919 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" Oct 30 17:34:31.917588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.917497 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb59754-4f88-4815-bc64-b53876d50b55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb59754-4f88-4815-bc64-b53876d50b55-plugins-config-volume\") pod \"4eb59754-4f88-4815-bc64-b53876d50b55\" (UID: \"4eb59754-4f88-4815-bc64-b53876d50b55\") " Oct 30 17:34:31.917588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.917556 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7d0036-8373-47e1-98f8-70c094b0e8d9-kube-api-access-xf4rx\" (UniqueName: \"kubernetes.io/projected/ea7d0036-8373-47e1-98f8-70c094b0e8d9-kube-api-access-xf4rx\") pod \"ea7d0036-8373-47e1-98f8-70c094b0e8d9\" (UID: \"ea7d0036-8373-47e1-98f8-70c094b0e8d9\") " Oct 30 17:34:31.917935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.917623 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7d0036-8373-47e1-98f8-70c094b0e8d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7d0036-8373-47e1-98f8-70c094b0e8d9-plugins-config-volume\") pod \"ea7d0036-8373-47e1-98f8-70c094b0e8d9\" (UID: \"ea7d0036-8373-47e1-98f8-70c094b0e8d9\") " Oct 30 17:34:31.917935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.917654 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb59754-4f88-4815-bc64-b53876d50b55-kube-api-access-whcmf\" (UniqueName: \"kubernetes.io/projected/4eb59754-4f88-4815-bc64-b53876d50b55-kube-api-access-whcmf\") pod \"4eb59754-4f88-4815-bc64-b53876d50b55\" (UID: \"4eb59754-4f88-4815-bc64-b53876d50b55\") " Oct 30 17:34:31.919522 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.919495 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb59754-4f88-4815-bc64-b53876d50b55-plugins-config-volume" pod "4eb59754-4f88-4815-bc64-b53876d50b55" (UID: "4eb59754-4f88-4815-bc64-b53876d50b55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:31.920759 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.920729 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7d0036-8373-47e1-98f8-70c094b0e8d9-plugins-config-volume" pod "ea7d0036-8373-47e1-98f8-70c094b0e8d9" (UID: "ea7d0036-8373-47e1-98f8-70c094b0e8d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:31.921138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.921105 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb59754-4f88-4815-bc64-b53876d50b55-kube-api-access-whcmf" pod "4eb59754-4f88-4815-bc64-b53876d50b55" (UID: "4eb59754-4f88-4815-bc64-b53876d50b55"). InnerVolumeSpecName "kube-api-access-whcmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:31.922960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:31.922886 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7d0036-8373-47e1-98f8-70c094b0e8d9-kube-api-access-xf4rx" pod "ea7d0036-8373-47e1-98f8-70c094b0e8d9" (UID: "ea7d0036-8373-47e1-98f8-70c094b0e8d9"). InnerVolumeSpecName "kube-api-access-xf4rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:32.020421 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.020387 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf4rx\" (UniqueName: \"kubernetes.io/projected/ea7d0036-8373-47e1-98f8-70c094b0e8d9-kube-api-access-xf4rx\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:32.020421 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.020420 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea7d0036-8373-47e1-98f8-70c094b0e8d9-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:32.020855 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.020436 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whcmf\" (UniqueName: \"kubernetes.io/projected/4eb59754-4f88-4815-bc64-b53876d50b55-kube-api-access-whcmf\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:32.020855 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.020449 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb59754-4f88-4815-bc64-b53876d50b55-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:32.736484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.736442 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" event={"ID":"4eb59754-4f88-4815-bc64-b53876d50b55","Type":"ContainerDied","Data":"ab7e5f561acef04788d19c8b6d6333bc840cfb230abae5550207319415ebaa88"} Oct 30 17:34:32.736700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.736520 4828 scope.go:122] "RemoveContainer" containerID="bcfff059c4a372fed3f88fc7c7c96a6ccac030c345f3de029b20f9ad601103bb" Oct 30 17:34:32.736700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.736452 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8" Oct 30 17:34:32.739352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.738703 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" event={"ID":"ea7d0036-8373-47e1-98f8-70c094b0e8d9","Type":"ContainerDied","Data":"6f91db1cff8fc027adbb9b841b7cca918814723168e66d9e2e496a0ce717d88c"} Oct 30 17:34:32.739352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.738727 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr" Oct 30 17:34:32.748488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.748465 4828 scope.go:122] "RemoveContainer" containerID="6c8ed4dd4548809e57332a5260e4f03b002ec29a84d1626eaf67fe47838985bc" Oct 30 17:34:32.793666 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.793582 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr"] Oct 30 17:34:32.808348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.808307 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5d8947-7cf5b856cf-6mmdr"] Oct 30 17:34:32.833004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.832968 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8"] Oct 30 17:34:32.838542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:32.838509 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5d8947-5bd6fc59b4-4f5r8"] Oct 30 17:34:33.092863 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:33.092735 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:33.640595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:33.640558 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" Oct 30 17:34:34.493578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:34.493545 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb59754-4f88-4815-bc64-b53876d50b55" path="/var/lib/kubelet/pods/4eb59754-4f88-4815-bc64-b53876d50b55/volumes" Oct 30 17:34:34.494088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:34.494073 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7d0036-8373-47e1-98f8-70c094b0e8d9" path="/var/lib/kubelet/pods/ea7d0036-8373-47e1-98f8-70c094b0e8d9/volumes" Oct 30 17:34:34.653030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:34.652992 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" Oct 30 17:34:34.747582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:34.747483 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:35.661068 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:35.661036 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" Oct 30 17:34:37.052881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:37.052834 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:37.913838 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:34:37.913786 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:34:38.668858 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:38.668816 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:40.456504 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:40.453194 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:40.924665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:40.924622 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:41.818530 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:41.818496 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4"] Oct 30 17:34:41.819020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:41.818709 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" podUID="1de99d4d-0549-4795-8e4b-1d1554a315c4" containerName="maas-api" containerID="cri-o://68b26f7a56fcc6b2c8241cf66692dd47858deba8a6f40084c231f2abf9562b69" gracePeriod=30 Oct 30 17:34:41.921582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:41.921544 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:41.974803 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:41.974769 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd"] Oct 30 17:34:41.975030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:41.975006 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" podUID="9e87a986-6278-48a7-b32e-742f7efd3c05" containerName="payload-processing" containerID="cri-o://ed7b593f8f96f5b7481e5043acf1076588ee4dd22b907a3734f8a8fa05676529" gracePeriod=30 Oct 30 17:34:41.997130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:41.997031 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4"] Oct 30 17:34:41.997287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:41.997258 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" podUID="6b374e83-1270-47c2-9620-88d9c947646d" containerName="payload-pre-processing" containerID="cri-o://035e296a0ff89159590a3fc7a5cc958bea30fb90d29267afb3562d1f6a3f303b" gracePeriod=30 Oct 30 17:34:42.133262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.132049 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:42.153718 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.153689 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:34:42.282987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.282841 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:42.319889 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.319860 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" Oct 30 17:34:42.383978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.377634 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1de99d4d-0549-4795-8e4b-1d1554a315c4-kube-api-access-nbgff\" (UniqueName: \"kubernetes.io/projected/1de99d4d-0549-4795-8e4b-1d1554a315c4-kube-api-access-nbgff\") pod \"1de99d4d-0549-4795-8e4b-1d1554a315c4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " Oct 30 17:34:42.383978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.377777 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e87a986-6278-48a7-b32e-742f7efd3c05-kube-api-access-ftlch\" (UniqueName: \"kubernetes.io/projected/9e87a986-6278-48a7-b32e-742f7efd3c05-kube-api-access-ftlch\") pod \"9e87a986-6278-48a7-b32e-742f7efd3c05\" (UID: \"9e87a986-6278-48a7-b32e-742f7efd3c05\") " Oct 30 17:34:42.383978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.377809 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-maas-api-tls\") pod \"1de99d4d-0549-4795-8e4b-1d1554a315c4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " Oct 30 17:34:42.383978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.377936 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-metrics-certs\") pod \"1de99d4d-0549-4795-8e4b-1d1554a315c4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " Oct 30 17:34:42.383978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.377967 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1de99d4d-0549-4795-8e4b-1d1554a315c4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de99d4d-0549-4795-8e4b-1d1554a315c4-openshift-service-ca\") pod \"1de99d4d-0549-4795-8e4b-1d1554a315c4\" (UID: \"1de99d4d-0549-4795-8e4b-1d1554a315c4\") " Oct 30 17:34:42.383978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.378013 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e87a986-6278-48a7-b32e-742f7efd3c05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e87a986-6278-48a7-b32e-742f7efd3c05-plugins-config-volume\") pod \"9e87a986-6278-48a7-b32e-742f7efd3c05\" (UID: \"9e87a986-6278-48a7-b32e-742f7efd3c05\") " Oct 30 17:34:42.422926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.397155 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e87a986-6278-48a7-b32e-742f7efd3c05-plugins-config-volume" pod "9e87a986-6278-48a7-b32e-742f7efd3c05" (UID: "9e87a986-6278-48a7-b32e-742f7efd3c05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:42.530925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.530072 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:42.550190 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.550146 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de99d4d-0549-4795-8e4b-1d1554a315c4-openshift-service-ca" pod "1de99d4d-0549-4795-8e4b-1d1554a315c4" (UID: "1de99d4d-0549-4795-8e4b-1d1554a315c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:42.562969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.562920 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de99d4d-0549-4795-8e4b-1d1554a315c4-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:42.563174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.563159 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e87a986-6278-48a7-b32e-742f7efd3c05-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:42.571273 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.571154 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e87a986-6278-48a7-b32e-742f7efd3c05-kube-api-access-ftlch" pod "9e87a986-6278-48a7-b32e-742f7efd3c05" (UID: "9e87a986-6278-48a7-b32e-742f7efd3c05"). InnerVolumeSpecName "kube-api-access-ftlch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:42.572465 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.572433 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de99d4d-0549-4795-8e4b-1d1554a315c4-kube-api-access-nbgff" pod "1de99d4d-0549-4795-8e4b-1d1554a315c4" (UID: "1de99d4d-0549-4795-8e4b-1d1554a315c4"). InnerVolumeSpecName "kube-api-access-nbgff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:42.575156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.574425 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-maas-api-tls" pod "1de99d4d-0549-4795-8e4b-1d1554a315c4" (UID: "1de99d4d-0549-4795-8e4b-1d1554a315c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:42.575156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.574592 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-metrics-certs" pod "1de99d4d-0549-4795-8e4b-1d1554a315c4" (UID: "1de99d4d-0549-4795-8e4b-1d1554a315c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:34:42.592238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.592203 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" Oct 30 17:34:42.665017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.664919 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b374e83-1270-47c2-9620-88d9c947646d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b374e83-1270-47c2-9620-88d9c947646d-plugins-config-volume\") pod \"6b374e83-1270-47c2-9620-88d9c947646d\" (UID: \"6b374e83-1270-47c2-9620-88d9c947646d\") " Oct 30 17:34:42.665213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.665070 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b374e83-1270-47c2-9620-88d9c947646d-kube-api-access-zzntf\" (UniqueName: \"kubernetes.io/projected/6b374e83-1270-47c2-9620-88d9c947646d-kube-api-access-zzntf\") pod \"6b374e83-1270-47c2-9620-88d9c947646d\" (UID: \"6b374e83-1270-47c2-9620-88d9c947646d\") " Oct 30 17:34:42.665213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.665200 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b374e83-1270-47c2-9620-88d9c947646d-plugins-config-volume" pod "6b374e83-1270-47c2-9620-88d9c947646d" (UID: "6b374e83-1270-47c2-9620-88d9c947646d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:42.665707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.665680 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b374e83-1270-47c2-9620-88d9c947646d-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:42.665707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.665710 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbgff\" (UniqueName: \"kubernetes.io/projected/1de99d4d-0549-4795-8e4b-1d1554a315c4-kube-api-access-nbgff\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:42.665926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.665726 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftlch\" (UniqueName: \"kubernetes.io/projected/9e87a986-6278-48a7-b32e-742f7efd3c05-kube-api-access-ftlch\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:42.665926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.665746 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:42.665926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.665760 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1de99d4d-0549-4795-8e4b-1d1554a315c4-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:42.667440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.667412 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b374e83-1270-47c2-9620-88d9c947646d-kube-api-access-zzntf" pod "6b374e83-1270-47c2-9620-88d9c947646d" (UID: "6b374e83-1270-47c2-9620-88d9c947646d"). InnerVolumeSpecName "kube-api-access-zzntf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:42.694025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.693885 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:42.768958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.768591 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzntf\" (UniqueName: \"kubernetes.io/projected/6b374e83-1270-47c2-9620-88d9c947646d-kube-api-access-zzntf\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:42.849154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.849011 4828 generic.go:356] "Generic (PLEG): container finished" podID="1de99d4d-0549-4795-8e4b-1d1554a315c4" containerID="68b26f7a56fcc6b2c8241cf66692dd47858deba8a6f40084c231f2abf9562b69" exitCode=0 Oct 30 17:34:42.849571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.849200 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" Oct 30 17:34:42.849571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.849251 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" event={"ID":"1de99d4d-0549-4795-8e4b-1d1554a315c4","Type":"ContainerDied","Data":"68b26f7a56fcc6b2c8241cf66692dd47858deba8a6f40084c231f2abf9562b69"} Oct 30 17:34:42.849571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.849304 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4" event={"ID":"1de99d4d-0549-4795-8e4b-1d1554a315c4","Type":"ContainerDied","Data":"ea222e432e4ab576a1c34579c7910106a70c158c35c3c92401150455d90b2d5e"} Oct 30 17:34:42.849571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.849335 4828 scope.go:122] "RemoveContainer" containerID="68b26f7a56fcc6b2c8241cf66692dd47858deba8a6f40084c231f2abf9562b69" Oct 30 17:34:42.849571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.849250 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:42.851042 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.851017 4828 generic.go:356] "Generic (PLEG): container finished" podID="6b374e83-1270-47c2-9620-88d9c947646d" containerID="035e296a0ff89159590a3fc7a5cc958bea30fb90d29267afb3562d1f6a3f303b" exitCode=0 Oct 30 17:34:42.851142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.851050 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" event={"ID":"6b374e83-1270-47c2-9620-88d9c947646d","Type":"ContainerDied","Data":"035e296a0ff89159590a3fc7a5cc958bea30fb90d29267afb3562d1f6a3f303b"} Oct 30 17:34:42.851142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.851081 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" Oct 30 17:34:42.851142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.851110 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4" event={"ID":"6b374e83-1270-47c2-9620-88d9c947646d","Type":"ContainerDied","Data":"f5c302143902d7ffb54c19ca70987d24bd702d7fdb6c9e05930b2982e2832cf8"} Oct 30 17:34:42.852749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.852723 4828 generic.go:356] "Generic (PLEG): container finished" podID="9e87a986-6278-48a7-b32e-742f7efd3c05" containerID="ed7b593f8f96f5b7481e5043acf1076588ee4dd22b907a3734f8a8fa05676529" exitCode=0 Oct 30 17:34:42.852872 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.852835 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" event={"ID":"9e87a986-6278-48a7-b32e-742f7efd3c05","Type":"ContainerDied","Data":"ed7b593f8f96f5b7481e5043acf1076588ee4dd22b907a3734f8a8fa05676529"} Oct 30 17:34:42.852967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.852876 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" event={"ID":"9e87a986-6278-48a7-b32e-742f7efd3c05","Type":"ContainerDied","Data":"757f5181e64ecd7071c97c34d504eb6809a24b892c3aaee9343186cb12789d67"} Oct 30 17:34:42.853038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.852990 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd" Oct 30 17:34:42.864804 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.864785 4828 scope.go:122] "RemoveContainer" containerID="68b26f7a56fcc6b2c8241cf66692dd47858deba8a6f40084c231f2abf9562b69" Oct 30 17:34:42.865109 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:34:42.865089 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b26f7a56fcc6b2c8241cf66692dd47858deba8a6f40084c231f2abf9562b69\": container with ID starting with 68b26f7a56fcc6b2c8241cf66692dd47858deba8a6f40084c231f2abf9562b69 not found: ID does not exist" containerID="68b26f7a56fcc6b2c8241cf66692dd47858deba8a6f40084c231f2abf9562b69" Oct 30 17:34:42.865183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.865117 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b26f7a56fcc6b2c8241cf66692dd47858deba8a6f40084c231f2abf9562b69"} err="failed to get container status \"68b26f7a56fcc6b2c8241cf66692dd47858deba8a6f40084c231f2abf9562b69\": rpc error: code = NotFound desc = could not find container \"68b26f7a56fcc6b2c8241cf66692dd47858deba8a6f40084c231f2abf9562b69\": container with ID starting with 68b26f7a56fcc6b2c8241cf66692dd47858deba8a6f40084c231f2abf9562b69 not found: ID does not exist" Oct 30 17:34:42.865183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.865134 4828 scope.go:122] "RemoveContainer" containerID="035e296a0ff89159590a3fc7a5cc958bea30fb90d29267afb3562d1f6a3f303b" Oct 30 17:34:42.878462 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.878430 4828 scope.go:122] "RemoveContainer" containerID="035e296a0ff89159590a3fc7a5cc958bea30fb90d29267afb3562d1f6a3f303b" Oct 30 17:34:42.878825 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:34:42.878795 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035e296a0ff89159590a3fc7a5cc958bea30fb90d29267afb3562d1f6a3f303b\": container with ID starting with 035e296a0ff89159590a3fc7a5cc958bea30fb90d29267afb3562d1f6a3f303b not found: ID does not exist" containerID="035e296a0ff89159590a3fc7a5cc958bea30fb90d29267afb3562d1f6a3f303b" Oct 30 17:34:42.878960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.878836 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035e296a0ff89159590a3fc7a5cc958bea30fb90d29267afb3562d1f6a3f303b"} err="failed to get container status \"035e296a0ff89159590a3fc7a5cc958bea30fb90d29267afb3562d1f6a3f303b\": rpc error: code = NotFound desc = could not find container \"035e296a0ff89159590a3fc7a5cc958bea30fb90d29267afb3562d1f6a3f303b\": container with ID starting with 035e296a0ff89159590a3fc7a5cc958bea30fb90d29267afb3562d1f6a3f303b not found: ID does not exist" Oct 30 17:34:42.878960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.878865 4828 scope.go:122] "RemoveContainer" containerID="ed7b593f8f96f5b7481e5043acf1076588ee4dd22b907a3734f8a8fa05676529" Oct 30 17:34:42.889775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.889753 4828 scope.go:122] "RemoveContainer" containerID="ed7b593f8f96f5b7481e5043acf1076588ee4dd22b907a3734f8a8fa05676529" Oct 30 17:34:42.890138 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:34:42.890117 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7b593f8f96f5b7481e5043acf1076588ee4dd22b907a3734f8a8fa05676529\": container with ID starting with ed7b593f8f96f5b7481e5043acf1076588ee4dd22b907a3734f8a8fa05676529 not found: ID does not exist" containerID="ed7b593f8f96f5b7481e5043acf1076588ee4dd22b907a3734f8a8fa05676529" Oct 30 17:34:42.890211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.890146 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7b593f8f96f5b7481e5043acf1076588ee4dd22b907a3734f8a8fa05676529"} err="failed to get container status \"ed7b593f8f96f5b7481e5043acf1076588ee4dd22b907a3734f8a8fa05676529\": rpc error: code = NotFound desc = could not find container \"ed7b593f8f96f5b7481e5043acf1076588ee4dd22b907a3734f8a8fa05676529\": container with ID starting with ed7b593f8f96f5b7481e5043acf1076588ee4dd22b907a3734f8a8fa05676529 not found: ID does not exist" Oct 30 17:34:42.907768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.907734 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd"] Oct 30 17:34:42.912253 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.912222 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3260f8-7fbd6d6858-d2knd"] Oct 30 17:34:42.947358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.947261 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4"] Oct 30 17:34:42.951543 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.951513 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3260f8-555689ff99-cpgj4"] Oct 30 17:34:42.987366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.987327 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4"] Oct 30 17:34:42.991631 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:42.991584 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3260f8-56c544c6f9-qtmj4"] Oct 30 17:34:43.187480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:43.187440 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:44.256764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:44.256729 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:44.502271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:44.502186 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1de99d4d-0549-4795-8e4b-1d1554a315c4" path="/var/lib/kubelet/pods/1de99d4d-0549-4795-8e4b-1d1554a315c4/volumes" Oct 30 17:34:44.505882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:44.502855 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b374e83-1270-47c2-9620-88d9c947646d" path="/var/lib/kubelet/pods/6b374e83-1270-47c2-9620-88d9c947646d/volumes" Oct 30 17:34:44.505882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:44.504497 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e87a986-6278-48a7-b32e-742f7efd3c05" path="/var/lib/kubelet/pods/9e87a986-6278-48a7-b32e-742f7efd3c05/volumes" Oct 30 17:34:44.873566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:44.873529 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:45.089595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:45.089555 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:45.248048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:45.247954 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:45.516657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:45.516618 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:47.127554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:47.127517 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:48.638130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:48.638091 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:48.838921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:48.838868 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.200910 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg"] Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202362 4828 cpu_manager.go:414] "containerMap: removing container" podUID="1de99d4d-0549-4795-8e4b-1d1554a315c4" containerName="maas-api" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202389 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1de99d4d-0549-4795-8e4b-1d1554a315c4" containerName="maas-api" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202421 4828 cpu_manager.go:414] "containerMap: removing container" podUID="4eb59754-4f88-4815-bc64-b53876d50b55" containerName="payload-processing" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202429 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb59754-4f88-4815-bc64-b53876d50b55" containerName="payload-processing" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202459 4828 cpu_manager.go:414] "containerMap: removing container" podUID="ea7d0036-8373-47e1-98f8-70c094b0e8d9" containerName="payload-pre-processing" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202468 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7d0036-8373-47e1-98f8-70c094b0e8d9" containerName="payload-pre-processing" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202484 4828 cpu_manager.go:414] "containerMap: removing container" podUID="ea2c6df0-ed33-4fff-9482-6e3cb6f6071f" containerName="maas-api" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202492 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea2c6df0-ed33-4fff-9482-6e3cb6f6071f" containerName="maas-api" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202519 4828 cpu_manager.go:414] "containerMap: removing container" podUID="9e87a986-6278-48a7-b32e-742f7efd3c05" containerName="payload-processing" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202527 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e87a986-6278-48a7-b32e-742f7efd3c05" containerName="payload-processing" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202561 4828 cpu_manager.go:414] "containerMap: removing container" podUID="6b374e83-1270-47c2-9620-88d9c947646d" containerName="payload-pre-processing" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202568 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b374e83-1270-47c2-9620-88d9c947646d" containerName="payload-pre-processing" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202752 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7d0036-8373-47e1-98f8-70c094b0e8d9" containerName="payload-pre-processing" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202762 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea2c6df0-ed33-4fff-9482-6e3cb6f6071f" containerName="maas-api" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202779 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e87a986-6278-48a7-b32e-742f7efd3c05" containerName="payload-processing" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202792 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b374e83-1270-47c2-9620-88d9c947646d" containerName="payload-pre-processing" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202805 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="1de99d4d-0549-4795-8e4b-1d1554a315c4" containerName="maas-api" Oct 30 17:34:49.207393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.202816 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb59754-4f88-4815-bc64-b53876d50b55" containerName="payload-processing" Oct 30 17:34:49.215858 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.215163 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" Oct 30 17:34:49.221675 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.221629 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg"] Oct 30 17:34:49.243030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.242965 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a949e7-dockercfg-ps4qd\"" Oct 30 17:34:49.243030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.242965 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a949e7\"" Oct 30 17:34:49.331233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.331194 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk"] Oct 30 17:34:49.335993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.335966 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" Oct 30 17:34:49.349778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.349741 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6e82e3-8647-478a-b1d9-9be624746f21-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg\" (UID: \"dd6e82e3-8647-478a-b1d9-9be624746f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" Oct 30 17:34:49.349991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.349805 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmnvs\" (UniqueName: \"kubernetes.io/projected/dd6e82e3-8647-478a-b1d9-9be624746f21-kube-api-access-mmnvs\") pod \"payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg\" (UID: \"dd6e82e3-8647-478a-b1d9-9be624746f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" Oct 30 17:34:49.361214 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.361184 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk"] Oct 30 17:34:49.451581 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.451547 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6e82e3-8647-478a-b1d9-9be624746f21-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg\" (UID: \"dd6e82e3-8647-478a-b1d9-9be624746f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" Oct 30 17:34:49.451581 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.451594 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e9873d-a365-4097-af99-34bca8251f98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk\" (UID: \"52e9873d-a365-4097-af99-34bca8251f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" Oct 30 17:34:49.451828 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.451637 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmnvs\" (UniqueName: \"kubernetes.io/projected/dd6e82e3-8647-478a-b1d9-9be624746f21-kube-api-access-mmnvs\") pod \"payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg\" (UID: \"dd6e82e3-8647-478a-b1d9-9be624746f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" Oct 30 17:34:49.451828 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.451681 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhc42\" (UniqueName: \"kubernetes.io/projected/52e9873d-a365-4097-af99-34bca8251f98-kube-api-access-xhc42\") pod \"payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk\" (UID: \"52e9873d-a365-4097-af99-34bca8251f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" Oct 30 17:34:49.452420 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.452394 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6e82e3-8647-478a-b1d9-9be624746f21-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg\" (UID: \"dd6e82e3-8647-478a-b1d9-9be624746f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" Oct 30 17:34:49.475940 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.475843 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmnvs\" (UniqueName: \"kubernetes.io/projected/dd6e82e3-8647-478a-b1d9-9be624746f21-kube-api-access-mmnvs\") pod \"payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg\" (UID: \"dd6e82e3-8647-478a-b1d9-9be624746f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" Oct 30 17:34:49.537613 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.537574 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" Oct 30 17:34:49.554456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.554419 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e9873d-a365-4097-af99-34bca8251f98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk\" (UID: \"52e9873d-a365-4097-af99-34bca8251f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" Oct 30 17:34:49.554618 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.554534 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhc42\" (UniqueName: \"kubernetes.io/projected/52e9873d-a365-4097-af99-34bca8251f98-kube-api-access-xhc42\") pod \"payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk\" (UID: \"52e9873d-a365-4097-af99-34bca8251f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" Oct 30 17:34:49.555549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.555498 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e9873d-a365-4097-af99-34bca8251f98-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk\" (UID: \"52e9873d-a365-4097-af99-34bca8251f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" Oct 30 17:34:49.581920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.578939 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhc42\" (UniqueName: \"kubernetes.io/projected/52e9873d-a365-4097-af99-34bca8251f98-kube-api-access-xhc42\") pod \"payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk\" (UID: \"52e9873d-a365-4097-af99-34bca8251f98\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" Oct 30 17:34:49.654923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.654618 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" Oct 30 17:34:49.786949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.781521 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:49.918607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.918566 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg"] Oct 30 17:34:49.925060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.925014 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" event={"ID":"dd6e82e3-8647-478a-b1d9-9be624746f21","Type":"ContainerStarted","Data":"306b81ec33993eb42f4f99a58a1eb3146be16ef72e1b3244afd3c95b24cfe611"} Oct 30 17:34:49.931166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:49.931137 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:50.010336 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:50.010311 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk"] Oct 30 17:34:50.193593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:50.193552 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:50.934757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:50.934713 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" event={"ID":"dd6e82e3-8647-478a-b1d9-9be624746f21","Type":"ContainerStarted","Data":"6bada3d2b00e31ad7e9543692ba46d5d175887c8f17264dd3e9930d27efa3bbb"} Oct 30 17:34:50.935268 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:50.934806 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" Oct 30 17:34:50.936963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:50.936891 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" event={"ID":"52e9873d-a365-4097-af99-34bca8251f98","Type":"ContainerStarted","Data":"54ed652033ea9881424fc2f935d202c3d5582b128a7793a395737ba0df4752be"} Oct 30 17:34:50.937120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:50.936988 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" event={"ID":"52e9873d-a365-4097-af99-34bca8251f98","Type":"ContainerStarted","Data":"8e33224c75172aa14a08e2d5f91577449ed4e20e51f84ce1abb35439c08bc6dd"} Oct 30 17:34:50.937650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:50.937628 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" Oct 30 17:34:50.987356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:50.980843 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" podStartSLOduration=1.980823903 podStartE2EDuration="1.980823903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:50.977780158 +0000 UTC m=+2868.623290337" watchObservedRunningTime="2026-10-30 17:34:50.980823903 +0000 UTC m=+2868.626334085" Oct 30 17:34:51.021917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:51.021851 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" podStartSLOduration=2.021833559 podStartE2EDuration="2.021833559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:34:51.021437053 +0000 UTC m=+2868.666947232" watchObservedRunningTime="2026-10-30 17:34:51.021833559 +0000 UTC m=+2868.667343731" Oct 30 17:34:51.443229 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:51.443187 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:51.646843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:51.643402 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:51.805407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:51.805370 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:52.084631 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:52.083592 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:52.305199 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:52.305113 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:52.454261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:52.454168 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:52.856258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:52.856223 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:53.036108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:53.036069 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:53.213995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:53.209020 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:53.225847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:53.225813 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv"] Oct 30 17:34:53.226139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:53.226108 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" containerID="cri-o://704c6269040e68ca3140a3c64e5cb2b7b04ddb2fad49d1be0643c28726fa36fc" gracePeriod=30 Oct 30 17:34:53.369924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:53.369288 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:54.456246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:54.456206 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:54.829108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:54.829072 4828 patch_prober.go:29] interesting pod/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:34:54.829285 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:54.829133 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:34:56.129136 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:56.129096 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:57.706936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:57.704976 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:58.526438 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.526412 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 17:34:58.667366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.667333 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-certs\") pod \"9edde938-9f51-43c7-9411-128b417a3edc\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " Oct 30 17:34:58.667536 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.667434 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9edde938-9f51-43c7-9411-128b417a3edc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edde938-9f51-43c7-9411-128b417a3edc-istio-podinfo\") pod \"9edde938-9f51-43c7-9411-128b417a3edc\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " Oct 30 17:34:58.667536 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.667469 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-credential-socket\") pod \"9edde938-9f51-43c7-9411-128b417a3edc\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " Oct 30 17:34:58.667622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.667582 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-kube-api-access-8lpgn\" (UniqueName: \"kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-kube-api-access-8lpgn\") pod \"9edde938-9f51-43c7-9411-128b417a3edc\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " Oct 30 17:34:58.667668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.667624 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-certs" pod "9edde938-9f51-43c7-9411-128b417a3edc" (UID: "9edde938-9f51-43c7-9411-128b417a3edc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:58.667724 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.667676 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-socket\") pod \"9edde938-9f51-43c7-9411-128b417a3edc\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " Oct 30 17:34:58.667770 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.667733 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9edde938-9f51-43c7-9411-128b417a3edc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edde938-9f51-43c7-9411-128b417a3edc-istiod-ca-cert\") pod \"9edde938-9f51-43c7-9411-128b417a3edc\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " Oct 30 17:34:58.667832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.667766 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-data\") pod \"9edde938-9f51-43c7-9411-128b417a3edc\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " Oct 30 17:34:58.667832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.667775 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-credential-socket" pod "9edde938-9f51-43c7-9411-128b417a3edc" (UID: "9edde938-9f51-43c7-9411-128b417a3edc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:58.667832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.667806 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-istio-token\" (UniqueName: \"kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-istio-token\") pod \"9edde938-9f51-43c7-9411-128b417a3edc\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " Oct 30 17:34:58.668026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.667850 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-envoy\") pod \"9edde938-9f51-43c7-9411-128b417a3edc\" (UID: \"9edde938-9f51-43c7-9411-128b417a3edc\") " Oct 30 17:34:58.668102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.668042 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-socket" pod "9edde938-9f51-43c7-9411-128b417a3edc" (UID: "9edde938-9f51-43c7-9411-128b417a3edc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:58.668181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.668095 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-data" pod "9edde938-9f51-43c7-9411-128b417a3edc" (UID: "9edde938-9f51-43c7-9411-128b417a3edc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:58.668181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.668145 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edde938-9f51-43c7-9411-128b417a3edc-istiod-ca-cert" pod "9edde938-9f51-43c7-9411-128b417a3edc" (UID: "9edde938-9f51-43c7-9411-128b417a3edc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:34:58.668533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.668510 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:58.668533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.668534 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9edde938-9f51-43c7-9411-128b417a3edc-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:58.668690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.668548 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:58.668690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.668560 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:58.668690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.668571 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:58.670045 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.670023 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9edde938-9f51-43c7-9411-128b417a3edc-istio-podinfo" pod "9edde938-9f51-43c7-9411-128b417a3edc" (UID: "9edde938-9f51-43c7-9411-128b417a3edc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:34:58.670143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.670107 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-kube-api-access-8lpgn" pod "9edde938-9f51-43c7-9411-128b417a3edc" (UID: "9edde938-9f51-43c7-9411-128b417a3edc"). InnerVolumeSpecName "kube-api-access-8lpgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:58.670352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.670334 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-istio-token" pod "9edde938-9f51-43c7-9411-128b417a3edc" (UID: "9edde938-9f51-43c7-9411-128b417a3edc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:34:58.670352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.670341 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-envoy" pod "9edde938-9f51-43c7-9411-128b417a3edc" (UID: "9edde938-9f51-43c7-9411-128b417a3edc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:34:58.769956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.769912 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9edde938-9f51-43c7-9411-128b417a3edc-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:58.769956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.769944 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpgn\" (UniqueName: \"kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-kube-api-access-8lpgn\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:58.769956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.769960 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9edde938-9f51-43c7-9411-128b417a3edc-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:58.770436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:58.769972 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9edde938-9f51-43c7-9411-128b417a3edc-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:34:59.042597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.042477 4828 generic.go:356] "Generic (PLEG): container finished" podID="9edde938-9f51-43c7-9411-128b417a3edc" containerID="704c6269040e68ca3140a3c64e5cb2b7b04ddb2fad49d1be0643c28726fa36fc" exitCode=0 Oct 30 17:34:59.042597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.042527 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" Oct 30 17:34:59.042597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.042545 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" event={"ID":"9edde938-9f51-43c7-9411-128b417a3edc","Type":"ContainerDied","Data":"704c6269040e68ca3140a3c64e5cb2b7b04ddb2fad49d1be0643c28726fa36fc"} Oct 30 17:34:59.042597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.042586 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv" event={"ID":"9edde938-9f51-43c7-9411-128b417a3edc","Type":"ContainerDied","Data":"daed19503e01908629f5dd37acdb9eded18b39faf03fabd27df194c0b8936947"} Oct 30 17:34:59.043083 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.042607 4828 scope.go:122] "RemoveContainer" containerID="704c6269040e68ca3140a3c64e5cb2b7b04ddb2fad49d1be0643c28726fa36fc" Oct 30 17:34:59.054477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.054455 4828 scope.go:122] "RemoveContainer" containerID="ed356183a53424b1347f9f31609e026763d711c1eb9fc71083ccac7f09c22347" Oct 30 17:34:59.064939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.064891 4828 scope.go:122] "RemoveContainer" containerID="704c6269040e68ca3140a3c64e5cb2b7b04ddb2fad49d1be0643c28726fa36fc" Oct 30 17:34:59.065270 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:34:59.065249 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704c6269040e68ca3140a3c64e5cb2b7b04ddb2fad49d1be0643c28726fa36fc\": container with ID starting with 704c6269040e68ca3140a3c64e5cb2b7b04ddb2fad49d1be0643c28726fa36fc not found: ID does not exist" containerID="704c6269040e68ca3140a3c64e5cb2b7b04ddb2fad49d1be0643c28726fa36fc" Oct 30 17:34:59.065346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.065284 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704c6269040e68ca3140a3c64e5cb2b7b04ddb2fad49d1be0643c28726fa36fc"} err="failed to get container status \"704c6269040e68ca3140a3c64e5cb2b7b04ddb2fad49d1be0643c28726fa36fc\": rpc error: code = NotFound desc = could not find container \"704c6269040e68ca3140a3c64e5cb2b7b04ddb2fad49d1be0643c28726fa36fc\": container with ID starting with 704c6269040e68ca3140a3c64e5cb2b7b04ddb2fad49d1be0643c28726fa36fc not found: ID does not exist" Oct 30 17:34:59.065346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.065310 4828 scope.go:122] "RemoveContainer" containerID="ed356183a53424b1347f9f31609e026763d711c1eb9fc71083ccac7f09c22347" Oct 30 17:34:59.065600 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:34:59.065582 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed356183a53424b1347f9f31609e026763d711c1eb9fc71083ccac7f09c22347\": container with ID starting with ed356183a53424b1347f9f31609e026763d711c1eb9fc71083ccac7f09c22347 not found: ID does not exist" containerID="ed356183a53424b1347f9f31609e026763d711c1eb9fc71083ccac7f09c22347" Oct 30 17:34:59.065644 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.065607 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed356183a53424b1347f9f31609e026763d711c1eb9fc71083ccac7f09c22347"} err="failed to get container status \"ed356183a53424b1347f9f31609e026763d711c1eb9fc71083ccac7f09c22347\": rpc error: code = NotFound desc = could not find container \"ed356183a53424b1347f9f31609e026763d711c1eb9fc71083ccac7f09c22347\": container with ID starting with ed356183a53424b1347f9f31609e026763d711c1eb9fc71083ccac7f09c22347 not found: ID does not exist" Oct 30 17:34:59.101481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.101442 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv"] Oct 30 17:34:59.112517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.112483 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c7b9df-openshift-default-5dfd6d94c-6glgv"] Oct 30 17:34:59.544833 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.544792 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:59.828418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.828333 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:34:59.994733 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:34:59.994689 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:00.120220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:00.120134 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2"] Oct 30 17:35:00.120441 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:00.120379 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" containerID="cri-o://1315ad1c6b1c90459df836d38610fdc28365b335f7458bfc08a7d3564ca15d16" gracePeriod=30 Oct 30 17:35:00.180368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:00.180325 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:00.346011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:00.343917 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:00.493578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:00.493506 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edde938-9f51-43c7-9411-128b417a3edc" path="/var/lib/kubelet/pods/9edde938-9f51-43c7-9411-128b417a3edc/volumes" Oct 30 17:35:00.680269 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:00.680225 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:00.971264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:00.971228 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:01.127605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.127561 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:01.423260 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.423221 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:35:01.423414 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.423282 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:35:01.788950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.786524 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:01.919447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.919409 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr"] Oct 30 17:35:01.920578 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.920553 4828 cpu_manager.go:414] "containerMap: removing container" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" Oct 30 17:35:01.920672 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.920581 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" Oct 30 17:35:01.920672 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.920635 4828 cpu_manager.go:414] "containerMap: removing container" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" Oct 30 17:35:01.920672 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.920644 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" Oct 30 17:35:01.920835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.920821 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" Oct 30 17:35:01.926555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.926533 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:01.948056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.948022 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr"] Oct 30 17:35:01.954114 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.954080 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8f92ef-serving-cert\"" Oct 30 17:35:01.988500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.988463 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" Oct 30 17:35:01.989063 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:01.989042 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" Oct 30 17:35:02.022792 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.022754 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:02.052517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.052423 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-metrics-certs\") pod \"maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:02.053044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.053003 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfcdp\" (UniqueName: \"kubernetes.io/projected/d0ee8b49-7ba2-485a-8f44-92f365ea306f-kube-api-access-zfcdp\") pod \"maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:02.053182 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.053119 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee8b49-7ba2-485a-8f44-92f365ea306f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:02.053236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.053186 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:02.154475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.154435 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfcdp\" (UniqueName: \"kubernetes.io/projected/d0ee8b49-7ba2-485a-8f44-92f365ea306f-kube-api-access-zfcdp\") pod \"maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:02.154661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.154492 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee8b49-7ba2-485a-8f44-92f365ea306f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:02.154661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.154627 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:02.154772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.154748 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-metrics-certs\") pod \"maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:02.155645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.155624 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee8b49-7ba2-485a-8f44-92f365ea306f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:02.158055 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.158035 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:02.158159 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.158032 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-metrics-certs\") pod \"maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:02.178987 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.178951 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfcdp\" (UniqueName: \"kubernetes.io/projected/d0ee8b49-7ba2-485a-8f44-92f365ea306f-kube-api-access-zfcdp\") pod \"maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:02.223931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.223297 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:02.248837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.248659 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:02.551565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.551530 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:02.612220 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:35:02.612191 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ee8b49_7ba2_485a_8f44_92f365ea306f.slice/crio-7d1db1393ab65641c07a0fdbddee8da84d4b1e488c82822eb710f5f447ba810b WatchSource:0}: Error finding container 7d1db1393ab65641c07a0fdbddee8da84d4b1e488c82822eb710f5f447ba810b: Status 404 returned error can't find the container with id 7d1db1393ab65641c07a0fdbddee8da84d4b1e488c82822eb710f5f447ba810b Oct 30 17:35:02.612568 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:02.612534 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr"] Oct 30 17:35:03.081388 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:03.081347 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" event={"ID":"d0ee8b49-7ba2-485a-8f44-92f365ea306f","Type":"ContainerStarted","Data":"7d1db1393ab65641c07a0fdbddee8da84d4b1e488c82822eb710f5f447ba810b"} Oct 30 17:35:03.967054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:03.966998 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:05.103365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.103327 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:05.105827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.105793 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" event={"ID":"d0ee8b49-7ba2-485a-8f44-92f365ea306f","Type":"ContainerStarted","Data":"62f5592054104d51e45c28c73215a0bd58380d27e55e334d7573fac57789907a"} Oct 30 17:35:05.106047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.105862 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:05.153480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.153434 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" podStartSLOduration=2.450255293 podStartE2EDuration="4.153419095s" totalImagesPullingTime="1.703163802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:05.149267279 +0000 UTC m=+2882.794777456" watchObservedRunningTime="2026-10-30 17:35:05.153419095 +0000 UTC m=+2882.798929275" Oct 30 17:35:05.517574 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.517529 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:35:05.550704 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.550663 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k"] Oct 30 17:35:05.555356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.555325 4828 cpu_manager.go:414] "containerMap: removing container" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" Oct 30 17:35:05.555356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.555354 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" Oct 30 17:35:05.555583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.555534 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" Oct 30 17:35:05.555583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.555550 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edde938-9f51-43c7-9411-128b417a3edc" containerName="istio-proxy" Oct 30 17:35:05.561037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.561009 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.583993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.583956 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-031223-openshift-default-dockercfg-pkngd\"" Oct 30 17:35:05.594617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.594583 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k"] Oct 30 17:35:05.646714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.645257 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-podinfo\") pod \"62228cc1-47cb-44f0-a0f4-9136e11b9263\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " Oct 30 17:35:05.646714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.645366 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62228cc1-47cb-44f0-a0f4-9136e11b9263-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62228cc1-47cb-44f0-a0f4-9136e11b9263-istiod-ca-cert\") pod \"62228cc1-47cb-44f0-a0f4-9136e11b9263\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " Oct 30 17:35:05.646714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.645452 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-envoy\") pod \"62228cc1-47cb-44f0-a0f4-9136e11b9263\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " Oct 30 17:35:05.646714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.645556 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-socket\") pod \"62228cc1-47cb-44f0-a0f4-9136e11b9263\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " Oct 30 17:35:05.646714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.645594 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-kube-api-access-hj7pq\" (UniqueName: \"kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-kube-api-access-hj7pq\") pod \"62228cc1-47cb-44f0-a0f4-9136e11b9263\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " Oct 30 17:35:05.646714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.645655 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-token\" (UniqueName: \"kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-token\") pod \"62228cc1-47cb-44f0-a0f4-9136e11b9263\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " Oct 30 17:35:05.646714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.645688 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-certs\") pod \"62228cc1-47cb-44f0-a0f4-9136e11b9263\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " Oct 30 17:35:05.646714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.645763 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-data\") pod \"62228cc1-47cb-44f0-a0f4-9136e11b9263\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " Oct 30 17:35:05.646714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.645934 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-credential-socket\") pod \"62228cc1-47cb-44f0-a0f4-9136e11b9263\" (UID: \"62228cc1-47cb-44f0-a0f4-9136e11b9263\") " Oct 30 17:35:05.648749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.647457 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-certs" pod "62228cc1-47cb-44f0-a0f4-9136e11b9263" (UID: "62228cc1-47cb-44f0-a0f4-9136e11b9263"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:05.648749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.648262 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:05.649233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.649038 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-data" pod "62228cc1-47cb-44f0-a0f4-9136e11b9263" (UID: "62228cc1-47cb-44f0-a0f4-9136e11b9263"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:05.650985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.649408 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-credential-socket" pod "62228cc1-47cb-44f0-a0f4-9136e11b9263" (UID: "62228cc1-47cb-44f0-a0f4-9136e11b9263"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:05.651684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.651513 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-socket" pod "62228cc1-47cb-44f0-a0f4-9136e11b9263" (UID: "62228cc1-47cb-44f0-a0f4-9136e11b9263"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:05.652239 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.651930 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62228cc1-47cb-44f0-a0f4-9136e11b9263-istiod-ca-cert" pod "62228cc1-47cb-44f0-a0f4-9136e11b9263" (UID: "62228cc1-47cb-44f0-a0f4-9136e11b9263"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.732499 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.750009 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-data\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.750101 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-envoy\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.750132 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-certs\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.750176 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-socket\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.750206 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c249be1c-0e0f-4380-917c-bb42dc4308c8-istiod-ca-cert\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.750259 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-podinfo\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.750304 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-token\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.750329 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-credential-socket\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.750449 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwsz7\" (UniqueName: \"kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-kube-api-access-pwsz7\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.750605 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.750625 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62228cc1-47cb-44f0-a0f4-9136e11b9263-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.750639 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:05.759033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.750650 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.854069 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-podinfo\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.854142 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-token\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.854171 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-credential-socket\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.854292 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwsz7\" (UniqueName: \"kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-kube-api-access-pwsz7\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.854452 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-data\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.854526 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-envoy\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.854559 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-certs\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.863232 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-socket\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.863303 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c249be1c-0e0f-4380-917c-bb42dc4308c8-istiod-ca-cert\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.864439 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c249be1c-0e0f-4380-917c-bb42dc4308c8-istiod-ca-cert\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.864967 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-envoy" pod "62228cc1-47cb-44f0-a0f4-9136e11b9263" (UID: "62228cc1-47cb-44f0-a0f4-9136e11b9263"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.865022 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-podinfo" pod "62228cc1-47cb-44f0-a0f4-9136e11b9263" (UID: "62228cc1-47cb-44f0-a0f4-9136e11b9263"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.865373 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-data\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.865432 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-certs\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.865861 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-kube-api-access-hj7pq" pod "62228cc1-47cb-44f0-a0f4-9136e11b9263" (UID: "62228cc1-47cb-44f0-a0f4-9136e11b9263"). InnerVolumeSpecName "kube-api-access-hj7pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.866152 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-credential-socket\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.866561 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-socket\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.867716 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-podinfo\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.868168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.867844 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-token" pod "62228cc1-47cb-44f0-a0f4-9136e11b9263" (UID: "62228cc1-47cb-44f0-a0f4-9136e11b9263"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:05.887841 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.886157 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-envoy\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.904813 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.897590 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:05.918134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.918052 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-token\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.919550 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.919519 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwsz7\" (UniqueName: \"kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-kube-api-access-pwsz7\") pod \"e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:05.965682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.965582 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:05.965682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.965619 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:05.965682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.965628 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj7pq\" (UniqueName: \"kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-kube-api-access-hj7pq\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:05.965682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:05.965637 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62228cc1-47cb-44f0-a0f4-9136e11b9263-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:06.118249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.118215 4828 generic.go:356] "Generic (PLEG): container finished" podID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerID="1315ad1c6b1c90459df836d38610fdc28365b335f7458bfc08a7d3564ca15d16" exitCode=0 Oct 30 17:35:06.118752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.118298 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" Oct 30 17:35:06.118752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.118297 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" event={"ID":"62228cc1-47cb-44f0-a0f4-9136e11b9263","Type":"ContainerDied","Data":"1315ad1c6b1c90459df836d38610fdc28365b335f7458bfc08a7d3564ca15d16"} Oct 30 17:35:06.118752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.118387 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2" event={"ID":"62228cc1-47cb-44f0-a0f4-9136e11b9263","Type":"ContainerDied","Data":"300d6f5982e41b2c30427a620083a44d0e497920edc209ea1d1ff2b7ec331ddb"} Oct 30 17:35:06.118752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.118424 4828 scope.go:122] "RemoveContainer" containerID="1315ad1c6b1c90459df836d38610fdc28365b335f7458bfc08a7d3564ca15d16" Oct 30 17:35:06.130981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.130923 4828 scope.go:122] "RemoveContainer" containerID="ab28cc0e98a1f87852848f975592b9fe22f0022d53eb5fcda8c58b0cbd423494" Oct 30 17:35:06.178636 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.178399 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:06.199939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.199886 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:06.204664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.204640 4828 scope.go:122] "RemoveContainer" containerID="1315ad1c6b1c90459df836d38610fdc28365b335f7458bfc08a7d3564ca15d16" Oct 30 17:35:06.205099 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:35:06.205073 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1315ad1c6b1c90459df836d38610fdc28365b335f7458bfc08a7d3564ca15d16\": container with ID starting with 1315ad1c6b1c90459df836d38610fdc28365b335f7458bfc08a7d3564ca15d16 not found: ID does not exist" containerID="1315ad1c6b1c90459df836d38610fdc28365b335f7458bfc08a7d3564ca15d16" Oct 30 17:35:06.205162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.205115 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1315ad1c6b1c90459df836d38610fdc28365b335f7458bfc08a7d3564ca15d16"} err="failed to get container status \"1315ad1c6b1c90459df836d38610fdc28365b335f7458bfc08a7d3564ca15d16\": rpc error: code = NotFound desc = could not find container \"1315ad1c6b1c90459df836d38610fdc28365b335f7458bfc08a7d3564ca15d16\": container with ID starting with 1315ad1c6b1c90459df836d38610fdc28365b335f7458bfc08a7d3564ca15d16 not found: ID does not exist" Oct 30 17:35:06.205162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.205139 4828 scope.go:122] "RemoveContainer" containerID="ab28cc0e98a1f87852848f975592b9fe22f0022d53eb5fcda8c58b0cbd423494" Oct 30 17:35:06.205436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.205418 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2"] Oct 30 17:35:06.205646 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:35:06.205600 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab28cc0e98a1f87852848f975592b9fe22f0022d53eb5fcda8c58b0cbd423494\": container with ID starting with ab28cc0e98a1f87852848f975592b9fe22f0022d53eb5fcda8c58b0cbd423494 not found: ID does not exist" containerID="ab28cc0e98a1f87852848f975592b9fe22f0022d53eb5fcda8c58b0cbd423494" Oct 30 17:35:06.205686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.205654 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab28cc0e98a1f87852848f975592b9fe22f0022d53eb5fcda8c58b0cbd423494"} err="failed to get container status \"ab28cc0e98a1f87852848f975592b9fe22f0022d53eb5fcda8c58b0cbd423494\": rpc error: code = NotFound desc = could not find container \"ab28cc0e98a1f87852848f975592b9fe22f0022d53eb5fcda8c58b0cbd423494\": container with ID starting with ab28cc0e98a1f87852848f975592b9fe22f0022d53eb5fcda8c58b0cbd423494 not found: ID does not exist" Oct 30 17:35:06.212824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.212786 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8a7e4a-openshift-default-685748496c-gkkw2"] Oct 30 17:35:06.495582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.495541 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" path="/var/lib/kubelet/pods/62228cc1-47cb-44f0-a0f4-9136e11b9263/volumes" Oct 30 17:35:06.534892 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.534865 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k"] Oct 30 17:35:06.537267 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:35:06.537241 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc249be1c_0e0f_4380_917c_bb42dc4308c8.slice/crio-284ee22ce46855300478fa0b3efade054f51d09d5e13a0cab30235c6eefa85d7 WatchSource:0}: Error finding container 284ee22ce46855300478fa0b3efade054f51d09d5e13a0cab30235c6eefa85d7: Status 404 returned error can't find the container with id 284ee22ce46855300478fa0b3efade054f51d09d5e13a0cab30235c6eefa85d7 Oct 30 17:35:06.540752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.540716 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:35:06.540879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.540787 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:35:06.540879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:06.540825 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:35:07.127506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:07.127471 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" event={"ID":"c249be1c-0e0f-4380-917c-bb42dc4308c8","Type":"ContainerStarted","Data":"ebcf70a9fc3860f0899fc2c6231c5abe4de1bf6de2371db824b856ce1a70a097"} Oct 30 17:35:07.128016 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:07.127530 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" event={"ID":"c249be1c-0e0f-4380-917c-bb42dc4308c8","Type":"ContainerStarted","Data":"284ee22ce46855300478fa0b3efade054f51d09d5e13a0cab30235c6eefa85d7"} Oct 30 17:35:07.178411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:07.178359 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" podStartSLOduration=2.178342299 podStartE2EDuration="2.178342299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:07.174115442 +0000 UTC m=+2884.819625622" watchObservedRunningTime="2026-10-30 17:35:07.178342299 +0000 UTC m=+2884.823852478" Oct 30 17:35:07.200571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:07.200533 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:07.206534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:07.206505 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:07.914074 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:35:07.914035 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:35:08.137178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:08.137145 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:08.138181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:08.138162 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:35:08.374602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:08.374563 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:09.233755 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:09.233713 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:09.392030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:09.391990 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:09.630382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:09.630351 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:09.806923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:09.801803 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:10.398923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:10.390091 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:11.124634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.124605 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:35:11.625376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.625293 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:11.766323 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.766272 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk"] Oct 30 17:35:11.767450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.767413 4828 cpu_manager.go:414] "containerMap: removing container" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" Oct 30 17:35:11.767450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.767440 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" Oct 30 17:35:11.767986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.767667 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="62228cc1-47cb-44f0-a0f4-9136e11b9263" containerName="istio-proxy" Oct 30 17:35:11.772337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.772317 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb"] Oct 30 17:35:11.772496 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.772473 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" Oct 30 17:35:11.779127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.779101 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" Oct 30 17:35:11.785067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.785035 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:11.789128 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.789106 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-031223\"" Oct 30 17:35:11.789760 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.789741 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-031223-dockercfg-459kl\"" Oct 30 17:35:11.809205 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.809175 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk"] Oct 30 17:35:11.815803 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:11.815769 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb"] Oct 30 17:35:12.035610 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.035568 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-266dd\" (UniqueName: \"kubernetes.io/projected/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-kube-api-access-266dd\") pod \"payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb\" (UID: \"c2b37307-2b2b-4e37-87f1-a1e898d17e3d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" Oct 30 17:35:12.035927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.035881 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h79rm\" (UniqueName: \"kubernetes.io/projected/8b51c1d9-8eb1-479a-9321-ee204c391050-kube-api-access-h79rm\") pod \"payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk\" (UID: \"8b51c1d9-8eb1-479a-9321-ee204c391050\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" Oct 30 17:35:12.036220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.036189 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b51c1d9-8eb1-479a-9321-ee204c391050-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk\" (UID: \"8b51c1d9-8eb1-479a-9321-ee204c391050\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" Oct 30 17:35:12.036633 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.036596 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb\" (UID: \"c2b37307-2b2b-4e37-87f1-a1e898d17e3d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" Oct 30 17:35:12.164253 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.164201 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266dd\" (UniqueName: \"kubernetes.io/projected/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-kube-api-access-266dd\") pod \"payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb\" (UID: \"c2b37307-2b2b-4e37-87f1-a1e898d17e3d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" Oct 30 17:35:12.164475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.164271 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h79rm\" (UniqueName: \"kubernetes.io/projected/8b51c1d9-8eb1-479a-9321-ee204c391050-kube-api-access-h79rm\") pod \"payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk\" (UID: \"8b51c1d9-8eb1-479a-9321-ee204c391050\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" Oct 30 17:35:12.164475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.164353 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b51c1d9-8eb1-479a-9321-ee204c391050-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk\" (UID: \"8b51c1d9-8eb1-479a-9321-ee204c391050\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" Oct 30 17:35:12.164612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.164494 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb\" (UID: \"c2b37307-2b2b-4e37-87f1-a1e898d17e3d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" Oct 30 17:35:12.165226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.165199 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b51c1d9-8eb1-479a-9321-ee204c391050-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk\" (UID: \"8b51c1d9-8eb1-479a-9321-ee204c391050\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" Oct 30 17:35:12.165485 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.165461 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb\" (UID: \"c2b37307-2b2b-4e37-87f1-a1e898d17e3d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" Oct 30 17:35:12.191333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.191294 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-266dd\" (UniqueName: \"kubernetes.io/projected/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-kube-api-access-266dd\") pod \"payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb\" (UID: \"c2b37307-2b2b-4e37-87f1-a1e898d17e3d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" Oct 30 17:35:12.195595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.195560 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h79rm\" (UniqueName: \"kubernetes.io/projected/8b51c1d9-8eb1-479a-9321-ee204c391050-kube-api-access-h79rm\") pod \"payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk\" (UID: \"8b51c1d9-8eb1-479a-9321-ee204c391050\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" Oct 30 17:35:12.392275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.392172 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" Oct 30 17:35:12.412317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.412276 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" Oct 30 17:35:12.714142 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:35:12.714107 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b51c1d9_8eb1_479a_9321_ee204c391050.slice/crio-973e5866e50bf50916abb735c42f269ac2a2b7dadcc5a40bd38c063e757358b2 WatchSource:0}: Error finding container 973e5866e50bf50916abb735c42f269ac2a2b7dadcc5a40bd38c063e757358b2: Status 404 returned error can't find the container with id 973e5866e50bf50916abb735c42f269ac2a2b7dadcc5a40bd38c063e757358b2 Oct 30 17:35:12.714614 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.714588 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk"] Oct 30 17:35:12.770033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.770001 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb"] Oct 30 17:35:12.776359 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:35:12.776315 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b37307_2b2b_4e37_87f1_a1e898d17e3d.slice/crio-fd6d27e523efccbf0ccc778e96980d33cd292274c63d6c5b2c7c19efea1f926e WatchSource:0}: Error finding container fd6d27e523efccbf0ccc778e96980d33cd292274c63d6c5b2c7c19efea1f926e: Status 404 returned error can't find the container with id fd6d27e523efccbf0ccc778e96980d33cd292274c63d6c5b2c7c19efea1f926e Oct 30 17:35:12.783783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:12.783725 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:13.221418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:13.221223 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" event={"ID":"c2b37307-2b2b-4e37-87f1-a1e898d17e3d","Type":"ContainerStarted","Data":"31ef76a3f7f17c16b5de96d52bc3d197b0ac5dd35f76c6ea99005138bd343ca9"} Oct 30 17:35:13.221418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:13.221338 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" event={"ID":"c2b37307-2b2b-4e37-87f1-a1e898d17e3d","Type":"ContainerStarted","Data":"fd6d27e523efccbf0ccc778e96980d33cd292274c63d6c5b2c7c19efea1f926e"} Oct 30 17:35:13.221418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:13.221367 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" Oct 30 17:35:13.223500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:13.223471 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" event={"ID":"8b51c1d9-8eb1-479a-9321-ee204c391050","Type":"ContainerStarted","Data":"bcaa21b66dfd173e3fdaa6b7486ac710cb28201d7636534f65fa39a747088e62"} Oct 30 17:35:13.223654 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:13.223528 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" Oct 30 17:35:13.223654 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:13.223546 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" event={"ID":"8b51c1d9-8eb1-479a-9321-ee204c391050","Type":"ContainerStarted","Data":"973e5866e50bf50916abb735c42f269ac2a2b7dadcc5a40bd38c063e757358b2"} Oct 30 17:35:13.270527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:13.270469 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" podStartSLOduration=2.270454586 podStartE2EDuration="2.270454586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:13.266997879 +0000 UTC m=+2890.912508059" watchObservedRunningTime="2026-10-30 17:35:13.270454586 +0000 UTC m=+2890.915964775" Oct 30 17:35:13.310393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:13.310332 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" podStartSLOduration=2.310318621 podStartE2EDuration="2.310318621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:35:13.30905353 +0000 UTC m=+2890.954563707" watchObservedRunningTime="2026-10-30 17:35:13.310318621 +0000 UTC m=+2890.955828801" Oct 30 17:35:14.639341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:14.639301 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:16.261000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.260962 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f"] Oct 30 17:35:16.261367 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.261264 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" podUID="c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e" containerName="maas-api" containerID="cri-o://307e2e44aa3bcf2a8af9e9ee1ade7398af2139e0573b2eaa02f6208e958dac76" gracePeriod=30 Oct 30 17:35:16.368288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.368246 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:16.671577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.671544 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:35:16.870929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.870825 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-metrics-certs\") pod \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " Oct 30 17:35:16.871087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.871036 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-kube-api-access-9zrjd\" (UniqueName: \"kubernetes.io/projected/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-kube-api-access-9zrjd\") pod \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " Oct 30 17:35:16.871131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.871083 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-maas-api-tls\") pod \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " Oct 30 17:35:16.871167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.871151 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-openshift-service-ca\") pod \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\" (UID: \"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e\") " Oct 30 17:35:16.872204 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.872177 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-openshift-service-ca" pod "c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e" (UID: "c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:16.880637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.880602 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-maas-api-tls" pod "c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e" (UID: "c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:16.881007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.880984 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-metrics-certs" pod "c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e" (UID: "c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:16.897305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.897109 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-kube-api-access-9zrjd" pod "c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e" (UID: "c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e"). InnerVolumeSpecName "kube-api-access-9zrjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:16.972747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.972704 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zrjd\" (UniqueName: \"kubernetes.io/projected/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-kube-api-access-9zrjd\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:16.972747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.972741 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:16.972747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.972752 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:16.972747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.972761 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:16.987641 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:16.987604 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:17.153358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:17.153255 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:17.257553 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:17.257523 4828 generic.go:356] "Generic (PLEG): container finished" podID="c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e" containerID="307e2e44aa3bcf2a8af9e9ee1ade7398af2139e0573b2eaa02f6208e958dac76" exitCode=0 Oct 30 17:35:17.257768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:17.257570 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" Oct 30 17:35:17.257768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:17.257604 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" event={"ID":"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e","Type":"ContainerDied","Data":"307e2e44aa3bcf2a8af9e9ee1ade7398af2139e0573b2eaa02f6208e958dac76"} Oct 30 17:35:17.257768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:17.257654 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f" event={"ID":"c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e","Type":"ContainerDied","Data":"3e33c95fa7142a4388dbb8e641215db3472b1b6cb816dc9382b26e839e0ea867"} Oct 30 17:35:17.257768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:17.257677 4828 scope.go:122] "RemoveContainer" containerID="307e2e44aa3bcf2a8af9e9ee1ade7398af2139e0573b2eaa02f6208e958dac76" Oct 30 17:35:17.269702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:17.269590 4828 scope.go:122] "RemoveContainer" containerID="307e2e44aa3bcf2a8af9e9ee1ade7398af2139e0573b2eaa02f6208e958dac76" Oct 30 17:35:17.270042 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:35:17.269967 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"307e2e44aa3bcf2a8af9e9ee1ade7398af2139e0573b2eaa02f6208e958dac76\": container with ID starting with 307e2e44aa3bcf2a8af9e9ee1ade7398af2139e0573b2eaa02f6208e958dac76 not found: ID does not exist" containerID="307e2e44aa3bcf2a8af9e9ee1ade7398af2139e0573b2eaa02f6208e958dac76" Oct 30 17:35:17.270042 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:17.270007 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"307e2e44aa3bcf2a8af9e9ee1ade7398af2139e0573b2eaa02f6208e958dac76"} err="failed to get container status \"307e2e44aa3bcf2a8af9e9ee1ade7398af2139e0573b2eaa02f6208e958dac76\": rpc error: code = NotFound desc = could not find container \"307e2e44aa3bcf2a8af9e9ee1ade7398af2139e0573b2eaa02f6208e958dac76\": container with ID starting with 307e2e44aa3bcf2a8af9e9ee1ade7398af2139e0573b2eaa02f6208e958dac76 not found: ID does not exist" Oct 30 17:35:17.337296 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:17.337251 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:17.337481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:17.337264 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f"] Oct 30 17:35:17.349213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:17.349178 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-11f480-67d6c4d5c9-zwl8f"] Oct 30 17:35:17.667720 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:17.667676 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:18.499677 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:18.499634 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e" path="/var/lib/kubelet/pods/c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e/volumes" Oct 30 17:35:18.954730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:18.954694 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:19.145331 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:19.145291 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:19.304548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:19.304509 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:19.728394 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:19.728298 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:19.893944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:19.893426 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:20.392358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:20.392320 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz"] Oct 30 17:35:20.418084 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:20.418046 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:20.600657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:20.600579 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:21.956516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:21.956434 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:23.606685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:23.606646 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:24.235609 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:24.235580 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" Oct 30 17:35:24.235810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:24.235645 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" Oct 30 17:35:25.180648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:25.180605 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:25.365155 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:25.365109 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:25.393691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:25.393635 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" podUID="3d9c5027-e57b-4b81-bdaf-3a8910099773" containerName="payload-processing" containerID="cri-o://0c4ffa6bddb053459d3b3cd9702fe2f99bd2460e1ddbb2c1d03a3d68725c4f88" gracePeriod=25 Oct 30 17:35:25.570667 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:25.570623 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:25.594837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:25.594800 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn"] Oct 30 17:35:25.595106 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:25.595078 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" podUID="7ab756a2-144f-4411-85db-dcb56009529f" containerName="istio-proxy" containerID="cri-o://5d2e9d2d0d1e8a3e6bfcb2dd13913477bee18a50ccbcb129da9689e4b660d326" gracePeriod=30 Oct 30 17:35:25.729359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:25.729332 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" Oct 30 17:35:25.767407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:25.767303 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:25.945923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:25.942534 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d9c5027-e57b-4b81-bdaf-3a8910099773-kube-api-access-m7f9j\" (UniqueName: \"kubernetes.io/projected/3d9c5027-e57b-4b81-bdaf-3a8910099773-kube-api-access-m7f9j\") pod \"3d9c5027-e57b-4b81-bdaf-3a8910099773\" (UID: \"3d9c5027-e57b-4b81-bdaf-3a8910099773\") " Oct 30 17:35:25.945923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:25.942769 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d9c5027-e57b-4b81-bdaf-3a8910099773-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9c5027-e57b-4b81-bdaf-3a8910099773-plugins-config-volume\") pod \"3d9c5027-e57b-4b81-bdaf-3a8910099773\" (UID: \"3d9c5027-e57b-4b81-bdaf-3a8910099773\") " Oct 30 17:35:25.945923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:25.944064 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9c5027-e57b-4b81-bdaf-3a8910099773-plugins-config-volume" pod "3d9c5027-e57b-4b81-bdaf-3a8910099773" (UID: "3d9c5027-e57b-4b81-bdaf-3a8910099773"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:25.980373 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:25.979656 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9c5027-e57b-4b81-bdaf-3a8910099773-kube-api-access-m7f9j" pod "3d9c5027-e57b-4b81-bdaf-3a8910099773" (UID: "3d9c5027-e57b-4b81-bdaf-3a8910099773"). InnerVolumeSpecName "kube-api-access-m7f9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:26.044771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:26.044682 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9c5027-e57b-4b81-bdaf-3a8910099773-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:26.044771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:26.044719 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7f9j\" (UniqueName: \"kubernetes.io/projected/3d9c5027-e57b-4b81-bdaf-3a8910099773-kube-api-access-m7f9j\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:26.345378 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:26.345294 4828 generic.go:356] "Generic (PLEG): container finished" podID="3d9c5027-e57b-4b81-bdaf-3a8910099773" containerID="0c4ffa6bddb053459d3b3cd9702fe2f99bd2460e1ddbb2c1d03a3d68725c4f88" exitCode=0 Oct 30 17:35:26.345378 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:26.345342 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" Oct 30 17:35:26.345884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:26.345380 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" event={"ID":"3d9c5027-e57b-4b81-bdaf-3a8910099773","Type":"ContainerDied","Data":"0c4ffa6bddb053459d3b3cd9702fe2f99bd2460e1ddbb2c1d03a3d68725c4f88"} Oct 30 17:35:26.345884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:26.345428 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz" event={"ID":"3d9c5027-e57b-4b81-bdaf-3a8910099773","Type":"ContainerDied","Data":"c62ab46e64c9298d32b9c64e84c97d13098d80a1c468f4d4a4b70468c168ab7f"} Oct 30 17:35:26.345884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:26.345448 4828 scope.go:122] "RemoveContainer" containerID="0c4ffa6bddb053459d3b3cd9702fe2f99bd2460e1ddbb2c1d03a3d68725c4f88" Oct 30 17:35:26.357017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:26.356998 4828 scope.go:122] "RemoveContainer" containerID="0c4ffa6bddb053459d3b3cd9702fe2f99bd2460e1ddbb2c1d03a3d68725c4f88" Oct 30 17:35:26.357298 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:35:26.357275 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4ffa6bddb053459d3b3cd9702fe2f99bd2460e1ddbb2c1d03a3d68725c4f88\": container with ID starting with 0c4ffa6bddb053459d3b3cd9702fe2f99bd2460e1ddbb2c1d03a3d68725c4f88 not found: ID does not exist" containerID="0c4ffa6bddb053459d3b3cd9702fe2f99bd2460e1ddbb2c1d03a3d68725c4f88" Oct 30 17:35:26.357382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:26.357304 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4ffa6bddb053459d3b3cd9702fe2f99bd2460e1ddbb2c1d03a3d68725c4f88"} err="failed to get container status \"0c4ffa6bddb053459d3b3cd9702fe2f99bd2460e1ddbb2c1d03a3d68725c4f88\": rpc error: code = NotFound desc = could not find container \"0c4ffa6bddb053459d3b3cd9702fe2f99bd2460e1ddbb2c1d03a3d68725c4f88\": container with ID starting with 0c4ffa6bddb053459d3b3cd9702fe2f99bd2460e1ddbb2c1d03a3d68725c4f88 not found: ID does not exist" Oct 30 17:35:26.447439 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:26.447402 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz"] Oct 30 17:35:26.458406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:26.458376 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a43843-578f8dd558-74rjz"] Oct 30 17:35:26.494276 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:26.494240 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d9c5027-e57b-4b81-bdaf-3a8910099773" path="/var/lib/kubelet/pods/3d9c5027-e57b-4b81-bdaf-3a8910099773/volumes" Oct 30 17:35:27.467927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:27.462594 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:28.055123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:28.055087 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:28.205887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:28.205852 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:28.403287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:28.403171 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:29.524738 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:29.524698 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:30.905547 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.905510 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:35:30.948694 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.948661 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-credential-socket\") pod \"7ab756a2-144f-4411-85db-dcb56009529f\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " Oct 30 17:35:30.948896 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.948704 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-envoy\") pod \"7ab756a2-144f-4411-85db-dcb56009529f\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " Oct 30 17:35:30.948896 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.948805 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-socket\") pod \"7ab756a2-144f-4411-85db-dcb56009529f\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " Oct 30 17:35:30.949043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.948894 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab756a2-144f-4411-85db-dcb56009529f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab756a2-144f-4411-85db-dcb56009529f-istiod-ca-cert\") pod \"7ab756a2-144f-4411-85db-dcb56009529f\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " Oct 30 17:35:30.949043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.948974 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-certs\") pod \"7ab756a2-144f-4411-85db-dcb56009529f\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " Oct 30 17:35:30.949043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.949008 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-data\") pod \"7ab756a2-144f-4411-85db-dcb56009529f\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " Oct 30 17:35:30.949207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.949037 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-credential-socket" pod "7ab756a2-144f-4411-85db-dcb56009529f" (UID: "7ab756a2-144f-4411-85db-dcb56009529f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.949207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.949045 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-istio-token\" (UniqueName: \"kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-istio-token\") pod \"7ab756a2-144f-4411-85db-dcb56009529f\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " Oct 30 17:35:30.949207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.949158 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ab756a2-144f-4411-85db-dcb56009529f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab756a2-144f-4411-85db-dcb56009529f-istio-podinfo\") pod \"7ab756a2-144f-4411-85db-dcb56009529f\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " Oct 30 17:35:30.949207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.949184 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-socket" pod "7ab756a2-144f-4411-85db-dcb56009529f" (UID: "7ab756a2-144f-4411-85db-dcb56009529f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.949416 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.949241 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-kube-api-access-kgsbw\" (UniqueName: \"kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-kube-api-access-kgsbw\") pod \"7ab756a2-144f-4411-85db-dcb56009529f\" (UID: \"7ab756a2-144f-4411-85db-dcb56009529f\") " Oct 30 17:35:30.949416 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.949360 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab756a2-144f-4411-85db-dcb56009529f-istiod-ca-cert" pod "7ab756a2-144f-4411-85db-dcb56009529f" (UID: "7ab756a2-144f-4411-85db-dcb56009529f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:30.949529 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.949460 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-data" pod "7ab756a2-144f-4411-85db-dcb56009529f" (UID: "7ab756a2-144f-4411-85db-dcb56009529f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.949529 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.949518 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-certs" pod "7ab756a2-144f-4411-85db-dcb56009529f" (UID: "7ab756a2-144f-4411-85db-dcb56009529f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.950062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.950035 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:30.950062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.950062 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:30.950257 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.950076 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab756a2-144f-4411-85db-dcb56009529f-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:30.950257 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.950088 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:30.950257 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.950103 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:30.951343 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.951322 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-envoy" pod "7ab756a2-144f-4411-85db-dcb56009529f" (UID: "7ab756a2-144f-4411-85db-dcb56009529f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:35:30.951737 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.951710 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ab756a2-144f-4411-85db-dcb56009529f-istio-podinfo" pod "7ab756a2-144f-4411-85db-dcb56009529f" (UID: "7ab756a2-144f-4411-85db-dcb56009529f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:35:30.952079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.952058 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-kube-api-access-kgsbw" pod "7ab756a2-144f-4411-85db-dcb56009529f" (UID: "7ab756a2-144f-4411-85db-dcb56009529f"). InnerVolumeSpecName "kube-api-access-kgsbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:30.952346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:30.952324 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-istio-token" pod "7ab756a2-144f-4411-85db-dcb56009529f" (UID: "7ab756a2-144f-4411-85db-dcb56009529f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:31.052121 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.052077 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.052121 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.052118 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab756a2-144f-4411-85db-dcb56009529f-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.052396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.052133 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgsbw\" (UniqueName: \"kubernetes.io/projected/7ab756a2-144f-4411-85db-dcb56009529f-kube-api-access-kgsbw\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.052396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.052149 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab756a2-144f-4411-85db-dcb56009529f-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:31.393869 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.393828 4828 generic.go:356] "Generic (PLEG): container finished" podID="7ab756a2-144f-4411-85db-dcb56009529f" containerID="5d2e9d2d0d1e8a3e6bfcb2dd13913477bee18a50ccbcb129da9689e4b660d326" exitCode=0 Oct 30 17:35:31.394062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.393919 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" Oct 30 17:35:31.394062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.393921 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" event={"ID":"7ab756a2-144f-4411-85db-dcb56009529f","Type":"ContainerDied","Data":"5d2e9d2d0d1e8a3e6bfcb2dd13913477bee18a50ccbcb129da9689e4b660d326"} Oct 30 17:35:31.394167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.394065 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn" event={"ID":"7ab756a2-144f-4411-85db-dcb56009529f","Type":"ContainerDied","Data":"a50667ac465df18dcbfdee746f246bd8d0bb0a7c90061b17b1596e7a8dc8c968"} Oct 30 17:35:31.394167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.394094 4828 scope.go:122] "RemoveContainer" containerID="5d2e9d2d0d1e8a3e6bfcb2dd13913477bee18a50ccbcb129da9689e4b660d326" Oct 30 17:35:31.407178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.407153 4828 scope.go:122] "RemoveContainer" containerID="5d2e9d2d0d1e8a3e6bfcb2dd13913477bee18a50ccbcb129da9689e4b660d326" Oct 30 17:35:31.408027 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:35:31.407998 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2e9d2d0d1e8a3e6bfcb2dd13913477bee18a50ccbcb129da9689e4b660d326\": container with ID starting with 5d2e9d2d0d1e8a3e6bfcb2dd13913477bee18a50ccbcb129da9689e4b660d326 not found: ID does not exist" containerID="5d2e9d2d0d1e8a3e6bfcb2dd13913477bee18a50ccbcb129da9689e4b660d326" Oct 30 17:35:31.408130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.408039 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2e9d2d0d1e8a3e6bfcb2dd13913477bee18a50ccbcb129da9689e4b660d326"} err="failed to get container status \"5d2e9d2d0d1e8a3e6bfcb2dd13913477bee18a50ccbcb129da9689e4b660d326\": rpc error: code = NotFound desc = could not find container \"5d2e9d2d0d1e8a3e6bfcb2dd13913477bee18a50ccbcb129da9689e4b660d326\": container with ID starting with 5d2e9d2d0d1e8a3e6bfcb2dd13913477bee18a50ccbcb129da9689e4b660d326 not found: ID does not exist" Oct 30 17:35:31.453033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.452994 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn"] Oct 30 17:35:31.463055 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.463022 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-11f480-openshift-default-86dc567cf-bnzdn"] Oct 30 17:35:31.964682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:31.964640 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:32.494082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:32.494049 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab756a2-144f-4411-85db-dcb56009529f" path="/var/lib/kubelet/pods/7ab756a2-144f-4411-85db-dcb56009529f/volumes" Oct 30 17:35:33.671037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:33.667828 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:35.465719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:35.465266 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:36.393597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:36.393557 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:36.546204 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:36.546155 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:37.168402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:37.168358 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:37.911867 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:35:37.911827 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:35:38.664837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:38.657247 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:39.145930 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:39.145873 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:39.304614 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:39.304578 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:40.511728 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:40.511690 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:42.247913 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:42.247797 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:44.032802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:44.032762 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:45.525043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:45.525008 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4"] Oct 30 17:35:45.525753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:45.525715 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" podUID="83781a17-2754-43ad-9832-6bf1700b0eeb" containerName="maas-api" containerID="cri-o://ab155f5c3760503f447353a4844f9009e55727ee4d9b8aa6a3cc7e93fa979947" gracePeriod=30 Oct 30 17:35:45.638178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:45.638053 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:45.767536 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:45.767491 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb"] Oct 30 17:35:45.768075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:45.768038 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" podUID="18f8f535-a840-4e43-8724-a408f694bf6c" containerName="payload-pre-processing" containerID="cri-o://d5cc840c1a00d18a6438c156e1cff83274d026b8d0c9ce35b398373a50d4f73f" gracePeriod=30 Oct 30 17:35:45.938442 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:45.938415 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:45.956332 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:45.956302 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:35:46.049950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.049911 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83781a17-2754-43ad-9832-6bf1700b0eeb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83781a17-2754-43ad-9832-6bf1700b0eeb-openshift-service-ca\") pod \"83781a17-2754-43ad-9832-6bf1700b0eeb\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " Oct 30 17:35:46.050143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.050089 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-maas-api-tls\") pod \"83781a17-2754-43ad-9832-6bf1700b0eeb\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " Oct 30 17:35:46.050231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.050148 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-metrics-certs\") pod \"83781a17-2754-43ad-9832-6bf1700b0eeb\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " Oct 30 17:35:46.050368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.050271 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83781a17-2754-43ad-9832-6bf1700b0eeb-kube-api-access-pcmgg\" (UniqueName: \"kubernetes.io/projected/83781a17-2754-43ad-9832-6bf1700b0eeb-kube-api-access-pcmgg\") pod \"83781a17-2754-43ad-9832-6bf1700b0eeb\" (UID: \"83781a17-2754-43ad-9832-6bf1700b0eeb\") " Oct 30 17:35:46.051222 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.051189 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83781a17-2754-43ad-9832-6bf1700b0eeb-openshift-service-ca" pod "83781a17-2754-43ad-9832-6bf1700b0eeb" (UID: "83781a17-2754-43ad-9832-6bf1700b0eeb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:46.053311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.053237 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-metrics-certs" pod "83781a17-2754-43ad-9832-6bf1700b0eeb" (UID: "83781a17-2754-43ad-9832-6bf1700b0eeb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:46.053580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.053543 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83781a17-2754-43ad-9832-6bf1700b0eeb-kube-api-access-pcmgg" pod "83781a17-2754-43ad-9832-6bf1700b0eeb" (UID: "83781a17-2754-43ad-9832-6bf1700b0eeb"). InnerVolumeSpecName "kube-api-access-pcmgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:46.053967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.053891 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-maas-api-tls" pod "83781a17-2754-43ad-9832-6bf1700b0eeb" (UID: "83781a17-2754-43ad-9832-6bf1700b0eeb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:46.124044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.122975 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:46.136517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.136479 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz"] Oct 30 17:35:46.136710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.136685 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" podUID="d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9" containerName="maas-api" containerID="cri-o://4a2b751e068e08f0279d080ba767bb54d72c80dfdfda993a6cafd3e15bcc6185" gracePeriod=30 Oct 30 17:35:46.152629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.152596 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:46.152884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.152866 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83781a17-2754-43ad-9832-6bf1700b0eeb-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:46.155709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.155628 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcmgg\" (UniqueName: \"kubernetes.io/projected/83781a17-2754-43ad-9832-6bf1700b0eeb-kube-api-access-pcmgg\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:46.155709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.155674 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83781a17-2754-43ad-9832-6bf1700b0eeb-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:46.342168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.334322 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:46.401976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.392281 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" Oct 30 17:35:46.429085 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.429049 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq"] Oct 30 17:35:46.436676 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.436374 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c"] Oct 30 17:35:46.539707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.538865 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:46.739975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.738880 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18f8f535-a840-4e43-8724-a408f694bf6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f8f535-a840-4e43-8724-a408f694bf6c-plugins-config-volume\") pod \"18f8f535-a840-4e43-8724-a408f694bf6c\" (UID: \"18f8f535-a840-4e43-8724-a408f694bf6c\") " Oct 30 17:35:46.739975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.739043 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18f8f535-a840-4e43-8724-a408f694bf6c-kube-api-access-8dnfn\" (UniqueName: \"kubernetes.io/projected/18f8f535-a840-4e43-8724-a408f694bf6c-kube-api-access-8dnfn\") pod \"18f8f535-a840-4e43-8724-a408f694bf6c\" (UID: \"18f8f535-a840-4e43-8724-a408f694bf6c\") " Oct 30 17:35:46.766229 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.764441 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:46.783095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.783060 4828 generic.go:356] "Generic (PLEG): container finished" podID="d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9" containerID="4a2b751e068e08f0279d080ba767bb54d72c80dfdfda993a6cafd3e15bcc6185" exitCode=0 Oct 30 17:35:46.792476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.785113 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f8f535-a840-4e43-8724-a408f694bf6c-kube-api-access-8dnfn" pod "18f8f535-a840-4e43-8724-a408f694bf6c" (UID: "18f8f535-a840-4e43-8724-a408f694bf6c"). InnerVolumeSpecName "kube-api-access-8dnfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:46.793502 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.793469 4828 generic.go:356] "Generic (PLEG): container finished" podID="83781a17-2754-43ad-9832-6bf1700b0eeb" containerID="ab155f5c3760503f447353a4844f9009e55727ee4d9b8aa6a3cc7e93fa979947" exitCode=0 Oct 30 17:35:46.802068 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.794035 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" Oct 30 17:35:46.815926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.804693 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" event={"ID":"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9","Type":"ContainerDied","Data":"4a2b751e068e08f0279d080ba767bb54d72c80dfdfda993a6cafd3e15bcc6185"} Oct 30 17:35:46.815926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.804857 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" event={"ID":"83781a17-2754-43ad-9832-6bf1700b0eeb","Type":"ContainerDied","Data":"ab155f5c3760503f447353a4844f9009e55727ee4d9b8aa6a3cc7e93fa979947"} Oct 30 17:35:46.815926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.804882 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4" event={"ID":"83781a17-2754-43ad-9832-6bf1700b0eeb","Type":"ContainerDied","Data":"5236ee8b8c4ac733a9156d6e55a364937a9eadea0e8eddbd48c5246f8ef971c8"} Oct 30 17:35:46.815926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.804932 4828 scope.go:122] "RemoveContainer" containerID="ab155f5c3760503f447353a4844f9009e55727ee4d9b8aa6a3cc7e93fa979947" Oct 30 17:35:46.815926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.813168 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18f8f535-a840-4e43-8724-a408f694bf6c-plugins-config-volume" pod "18f8f535-a840-4e43-8724-a408f694bf6c" (UID: "18f8f535-a840-4e43-8724-a408f694bf6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:46.842859 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.841807 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dnfn\" (UniqueName: \"kubernetes.io/projected/18f8f535-a840-4e43-8724-a408f694bf6c-kube-api-access-8dnfn\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:46.842859 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.841842 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f8f535-a840-4e43-8724-a408f694bf6c-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:46.925401 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.925374 4828 generic.go:356] "Generic (PLEG): container finished" podID="18f8f535-a840-4e43-8724-a408f694bf6c" containerID="d5cc840c1a00d18a6438c156e1cff83274d026b8d0c9ce35b398373a50d4f73f" exitCode=0 Oct 30 17:35:46.925568 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.925520 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" Oct 30 17:35:46.925847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.925550 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" event={"ID":"18f8f535-a840-4e43-8724-a408f694bf6c","Type":"ContainerDied","Data":"d5cc840c1a00d18a6438c156e1cff83274d026b8d0c9ce35b398373a50d4f73f"} Oct 30 17:35:46.925847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.925803 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb" event={"ID":"18f8f535-a840-4e43-8724-a408f694bf6c","Type":"ContainerDied","Data":"cd95ee0b4e5e01225c52bc31d97137e024c107544e0f067078cb7520736d8d53"} Oct 30 17:35:46.939265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.939236 4828 scope.go:122] "RemoveContainer" containerID="ab155f5c3760503f447353a4844f9009e55727ee4d9b8aa6a3cc7e93fa979947" Oct 30 17:35:46.940468 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:35:46.940282 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab155f5c3760503f447353a4844f9009e55727ee4d9b8aa6a3cc7e93fa979947\": container with ID starting with ab155f5c3760503f447353a4844f9009e55727ee4d9b8aa6a3cc7e93fa979947 not found: ID does not exist" containerID="ab155f5c3760503f447353a4844f9009e55727ee4d9b8aa6a3cc7e93fa979947" Oct 30 17:35:46.940468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.940328 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab155f5c3760503f447353a4844f9009e55727ee4d9b8aa6a3cc7e93fa979947"} err="failed to get container status \"ab155f5c3760503f447353a4844f9009e55727ee4d9b8aa6a3cc7e93fa979947\": rpc error: code = NotFound desc = could not find container \"ab155f5c3760503f447353a4844f9009e55727ee4d9b8aa6a3cc7e93fa979947\": container with ID starting with ab155f5c3760503f447353a4844f9009e55727ee4d9b8aa6a3cc7e93fa979947 not found: ID does not exist" Oct 30 17:35:46.940468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.940356 4828 scope.go:122] "RemoveContainer" containerID="d5cc840c1a00d18a6438c156e1cff83274d026b8d0c9ce35b398373a50d4f73f" Oct 30 17:35:46.959700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.959678 4828 scope.go:122] "RemoveContainer" containerID="d5cc840c1a00d18a6438c156e1cff83274d026b8d0c9ce35b398373a50d4f73f" Oct 30 17:35:46.960058 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:35:46.960031 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5cc840c1a00d18a6438c156e1cff83274d026b8d0c9ce35b398373a50d4f73f\": container with ID starting with d5cc840c1a00d18a6438c156e1cff83274d026b8d0c9ce35b398373a50d4f73f not found: ID does not exist" containerID="d5cc840c1a00d18a6438c156e1cff83274d026b8d0c9ce35b398373a50d4f73f" Oct 30 17:35:46.960221 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:46.960069 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5cc840c1a00d18a6438c156e1cff83274d026b8d0c9ce35b398373a50d4f73f"} err="failed to get container status \"d5cc840c1a00d18a6438c156e1cff83274d026b8d0c9ce35b398373a50d4f73f\": rpc error: code = NotFound desc = could not find container \"d5cc840c1a00d18a6438c156e1cff83274d026b8d0c9ce35b398373a50d4f73f\": container with ID starting with d5cc840c1a00d18a6438c156e1cff83274d026b8d0c9ce35b398373a50d4f73f not found: ID does not exist" Oct 30 17:35:47.013102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.012886 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4"] Oct 30 17:35:47.019844 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.019806 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4fb821-f7c79d4cf-2wkh4"] Oct 30 17:35:47.039280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.039236 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb"] Oct 30 17:35:47.048758 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.048699 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4fb821-74bd4876dc-h6vrb"] Oct 30 17:35:47.082953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.082143 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:35:47.109918 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.109716 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:47.147333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.147297 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-maas-api-tls\") pod \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " Oct 30 17:35:47.147522 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.147489 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-kube-api-access-g2c4g\" (UniqueName: \"kubernetes.io/projected/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-kube-api-access-g2c4g\") pod \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " Oct 30 17:35:47.147596 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.147580 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-metrics-certs\") pod \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " Oct 30 17:35:47.147657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.147614 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-openshift-service-ca\") pod \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\" (UID: \"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9\") " Oct 30 17:35:47.148055 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.148027 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-openshift-service-ca" pod "d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9" (UID: "d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:47.148241 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.148223 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:47.149686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.149665 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-maas-api-tls" pod "d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9" (UID: "d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:47.150067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.150035 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-kube-api-access-g2c4g" pod "d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9" (UID: "d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9"). InnerVolumeSpecName "kube-api-access-g2c4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:47.150762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.150591 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-metrics-certs" pod "d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9" (UID: "d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:47.249298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.249261 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:47.249298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.249293 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2c4g\" (UniqueName: \"kubernetes.io/projected/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-kube-api-access-g2c4g\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:47.249526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.249306 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:47.279464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.279361 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:47.446471 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.446263 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:47.616204 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.616106 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:47.769870 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.769828 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:47.937161 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.937075 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" Oct 30 17:35:47.937161 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.937101 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz" event={"ID":"d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9","Type":"ContainerDied","Data":"76403eb0bd5ceb3c4c085a3e86b279b981cbc7672ad3e0a5614fbf51d9c1cca5"} Oct 30 17:35:47.937394 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:47.937172 4828 scope.go:122] "RemoveContainer" containerID="4a2b751e068e08f0279d080ba767bb54d72c80dfdfda993a6cafd3e15bcc6185" Oct 30 17:35:48.000935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:48.000880 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz"] Oct 30 17:35:48.013234 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:48.013196 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8681b5-5c45999894-lxhpz"] Oct 30 17:35:48.229238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:48.229138 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:48.497668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:48.497595 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18f8f535-a840-4e43-8724-a408f694bf6c" path="/var/lib/kubelet/pods/18f8f535-a840-4e43-8724-a408f694bf6c/volumes" Oct 30 17:35:48.498207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:48.498189 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83781a17-2754-43ad-9832-6bf1700b0eeb" path="/var/lib/kubelet/pods/83781a17-2754-43ad-9832-6bf1700b0eeb/volumes" Oct 30 17:35:48.498625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:48.498611 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9" path="/var/lib/kubelet/pods/d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9/volumes" Oct 30 17:35:48.608044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:48.606392 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:35:48.608044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:48.606464 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:35:49.396801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:49.396721 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:51.368620 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.368569 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:51.430292 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.430247 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" podUID="32a43570-25ea-4738-9109-85aaf132547d" containerName="payload-processing" containerID="cri-o://493640af5ea80450e1df546ec795e81378edbccc2c6cb0a4d049fc3b496be0b8" gracePeriod=25 Oct 30 17:35:51.441255 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.437821 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" podUID="e84d9cf0-b5f6-4e24-b189-4e80e341e9f9" containerName="payload-pre-processing" containerID="cri-o://0c4e042fd4fc2d4648f994adac711aff10142290b4b6b10a322f6e3fd9234bb7" gracePeriod=25 Oct 30 17:35:51.745462 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.745429 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" Oct 30 17:35:51.774594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.774556 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" Oct 30 17:35:51.847404 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.847373 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-plugins-config-volume\") pod \"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9\" (UID: \"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9\") " Oct 30 17:35:51.847577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.847481 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a43570-25ea-4738-9109-85aaf132547d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a43570-25ea-4738-9109-85aaf132547d-plugins-config-volume\") pod \"32a43570-25ea-4738-9109-85aaf132547d\" (UID: \"32a43570-25ea-4738-9109-85aaf132547d\") " Oct 30 17:35:51.847577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.847513 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-kube-api-access-lp8wr\" (UniqueName: \"kubernetes.io/projected/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-kube-api-access-lp8wr\") pod \"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9\" (UID: \"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9\") " Oct 30 17:35:51.847577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.847555 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a43570-25ea-4738-9109-85aaf132547d-kube-api-access-b2cg2\" (UniqueName: \"kubernetes.io/projected/32a43570-25ea-4738-9109-85aaf132547d-kube-api-access-b2cg2\") pod \"32a43570-25ea-4738-9109-85aaf132547d\" (UID: \"32a43570-25ea-4738-9109-85aaf132547d\") " Oct 30 17:35:51.847859 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.847832 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-plugins-config-volume" pod "e84d9cf0-b5f6-4e24-b189-4e80e341e9f9" (UID: "e84d9cf0-b5f6-4e24-b189-4e80e341e9f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:51.847991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.847866 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a43570-25ea-4738-9109-85aaf132547d-plugins-config-volume" pod "32a43570-25ea-4738-9109-85aaf132547d" (UID: "32a43570-25ea-4738-9109-85aaf132547d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:51.848352 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.848327 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32a43570-25ea-4738-9109-85aaf132547d-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:51.848468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.848355 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:51.849879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.849852 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a43570-25ea-4738-9109-85aaf132547d-kube-api-access-b2cg2" pod "32a43570-25ea-4738-9109-85aaf132547d" (UID: "32a43570-25ea-4738-9109-85aaf132547d"). InnerVolumeSpecName "kube-api-access-b2cg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:51.851309 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.851288 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-kube-api-access-lp8wr" pod "e84d9cf0-b5f6-4e24-b189-4e80e341e9f9" (UID: "e84d9cf0-b5f6-4e24-b189-4e80e341e9f9"). InnerVolumeSpecName "kube-api-access-lp8wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:51.949649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.949561 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp8wr\" (UniqueName: \"kubernetes.io/projected/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9-kube-api-access-lp8wr\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:51.949649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.949598 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2cg2\" (UniqueName: \"kubernetes.io/projected/32a43570-25ea-4738-9109-85aaf132547d-kube-api-access-b2cg2\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:51.977227 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.977192 4828 generic.go:356] "Generic (PLEG): container finished" podID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerID="fa94b91391edcffb19b35439803dad5a28e1676757939c06affb8413468de3a8" exitCode=0 Oct 30 17:35:51.977431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.977258 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" event={"ID":"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5","Type":"ContainerDied","Data":"fa94b91391edcffb19b35439803dad5a28e1676757939c06affb8413468de3a8"} Oct 30 17:35:51.977824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.977805 4828 scope.go:122] "RemoveContainer" containerID="fa94b91391edcffb19b35439803dad5a28e1676757939c06affb8413468de3a8" Oct 30 17:35:51.979295 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.979273 4828 generic.go:356] "Generic (PLEG): container finished" podID="e84d9cf0-b5f6-4e24-b189-4e80e341e9f9" containerID="0c4e042fd4fc2d4648f994adac711aff10142290b4b6b10a322f6e3fd9234bb7" exitCode=0 Oct 30 17:35:51.979456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.979373 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" Oct 30 17:35:51.979540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.979442 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" event={"ID":"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9","Type":"ContainerDied","Data":"0c4e042fd4fc2d4648f994adac711aff10142290b4b6b10a322f6e3fd9234bb7"} Oct 30 17:35:51.979540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.979510 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c" event={"ID":"e84d9cf0-b5f6-4e24-b189-4e80e341e9f9","Type":"ContainerDied","Data":"5c5c3aac454fa2c0295ec285751a6d47d567e9b1b82de8b02a45cb3c4de785fd"} Oct 30 17:35:51.980279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.979541 4828 scope.go:122] "RemoveContainer" containerID="0c4e042fd4fc2d4648f994adac711aff10142290b4b6b10a322f6e3fd9234bb7" Oct 30 17:35:51.980279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.979930 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:35:51.980279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.980018 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:35:51.980279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.980066 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:35:51.984284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.984157 4828 generic.go:356] "Generic (PLEG): container finished" podID="32a43570-25ea-4738-9109-85aaf132547d" containerID="493640af5ea80450e1df546ec795e81378edbccc2c6cb0a4d049fc3b496be0b8" exitCode=0 Oct 30 17:35:51.984367 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.984346 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" event={"ID":"32a43570-25ea-4738-9109-85aaf132547d","Type":"ContainerDied","Data":"493640af5ea80450e1df546ec795e81378edbccc2c6cb0a4d049fc3b496be0b8"} Oct 30 17:35:51.984428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.984397 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" event={"ID":"32a43570-25ea-4738-9109-85aaf132547d","Type":"ContainerDied","Data":"613739da28811c633cfcd4f758d2b65d6673755fe6c277d1942a5668c34ab937"} Oct 30 17:35:51.984489 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.984470 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq" Oct 30 17:35:51.999689 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:51.999598 4828 scope.go:122] "RemoveContainer" containerID="0c4e042fd4fc2d4648f994adac711aff10142290b4b6b10a322f6e3fd9234bb7" Oct 30 17:35:52.000013 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:35:51.999988 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4e042fd4fc2d4648f994adac711aff10142290b4b6b10a322f6e3fd9234bb7\": container with ID starting with 0c4e042fd4fc2d4648f994adac711aff10142290b4b6b10a322f6e3fd9234bb7 not found: ID does not exist" containerID="0c4e042fd4fc2d4648f994adac711aff10142290b4b6b10a322f6e3fd9234bb7" Oct 30 17:35:52.000111 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:52.000019 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4e042fd4fc2d4648f994adac711aff10142290b4b6b10a322f6e3fd9234bb7"} err="failed to get container status \"0c4e042fd4fc2d4648f994adac711aff10142290b4b6b10a322f6e3fd9234bb7\": rpc error: code = NotFound desc = could not find container \"0c4e042fd4fc2d4648f994adac711aff10142290b4b6b10a322f6e3fd9234bb7\": container with ID starting with 0c4e042fd4fc2d4648f994adac711aff10142290b4b6b10a322f6e3fd9234bb7 not found: ID does not exist" Oct 30 17:35:52.000111 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:52.000038 4828 scope.go:122] "RemoveContainer" containerID="493640af5ea80450e1df546ec795e81378edbccc2c6cb0a4d049fc3b496be0b8" Oct 30 17:35:52.012272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:52.012242 4828 scope.go:122] "RemoveContainer" containerID="493640af5ea80450e1df546ec795e81378edbccc2c6cb0a4d049fc3b496be0b8" Oct 30 17:35:52.012645 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:35:52.012612 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493640af5ea80450e1df546ec795e81378edbccc2c6cb0a4d049fc3b496be0b8\": container with ID starting with 493640af5ea80450e1df546ec795e81378edbccc2c6cb0a4d049fc3b496be0b8 not found: ID does not exist" containerID="493640af5ea80450e1df546ec795e81378edbccc2c6cb0a4d049fc3b496be0b8" Oct 30 17:35:52.012747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:52.012657 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493640af5ea80450e1df546ec795e81378edbccc2c6cb0a4d049fc3b496be0b8"} err="failed to get container status \"493640af5ea80450e1df546ec795e81378edbccc2c6cb0a4d049fc3b496be0b8\": rpc error: code = NotFound desc = could not find container \"493640af5ea80450e1df546ec795e81378edbccc2c6cb0a4d049fc3b496be0b8\": container with ID starting with 493640af5ea80450e1df546ec795e81378edbccc2c6cb0a4d049fc3b496be0b8 not found: ID does not exist" Oct 30 17:35:52.073261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:52.067733 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c"] Oct 30 17:35:52.073733 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:52.073699 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8f92ef-5d87d947bc-g875c"] Oct 30 17:35:52.109755 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:52.109710 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq"] Oct 30 17:35:52.117786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:52.117755 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8f92ef-5b8b5bfd4-przhq"] Oct 30 17:35:52.494079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:52.494047 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a43570-25ea-4738-9109-85aaf132547d" path="/var/lib/kubelet/pods/32a43570-25ea-4738-9109-85aaf132547d/volumes" Oct 30 17:35:52.494530 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:52.494514 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84d9cf0-b5f6-4e24-b189-4e80e341e9f9" path="/var/lib/kubelet/pods/e84d9cf0-b5f6-4e24-b189-4e80e341e9f9/volumes" Oct 30 17:35:52.953437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:52.953403 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:53.018353 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:53.018313 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" event={"ID":"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5","Type":"ContainerStarted","Data":"8fef74fd88446e6f699c459cc8a25bfcfad6758904ae65ea299fda89c545356b"} Oct 30 17:35:53.168145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:53.168107 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:35:53.168330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:53.168307 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 17:35:53.168372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:53.168354 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 17:35:53.406543 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:53.406494 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:53.563998 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:53.563962 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:53.798924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:53.796400 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:54.167866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.167779 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 17:35:54.167866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.167853 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 17:35:54.462779 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.462669 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n"] Oct 30 17:35:54.463008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.462983 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" podUID="21e851e1-0779-48c0-9e4b-2236addceb52" containerName="maas-api" containerID="cri-o://83b7f91715d3995db6fd63d6b6578410638caf8d4c92ce8c95ba1636d89e67b7" gracePeriod=30 Oct 30 17:35:54.600672 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.600633 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:54.823190 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.823165 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:35:54.955931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.953483 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-metrics-certs\") pod \"21e851e1-0779-48c0-9e4b-2236addceb52\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " Oct 30 17:35:54.955931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.953615 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e851e1-0779-48c0-9e4b-2236addceb52-kube-api-access-dmdjs\" (UniqueName: \"kubernetes.io/projected/21e851e1-0779-48c0-9e4b-2236addceb52-kube-api-access-dmdjs\") pod \"21e851e1-0779-48c0-9e4b-2236addceb52\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " Oct 30 17:35:54.955931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.954035 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-maas-api-tls\") pod \"21e851e1-0779-48c0-9e4b-2236addceb52\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " Oct 30 17:35:54.955931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.954096 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e851e1-0779-48c0-9e4b-2236addceb52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e851e1-0779-48c0-9e4b-2236addceb52-openshift-service-ca\") pod \"21e851e1-0779-48c0-9e4b-2236addceb52\" (UID: \"21e851e1-0779-48c0-9e4b-2236addceb52\") " Oct 30 17:35:54.955931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.955367 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e851e1-0779-48c0-9e4b-2236addceb52-openshift-service-ca" pod "21e851e1-0779-48c0-9e4b-2236addceb52" (UID: "21e851e1-0779-48c0-9e4b-2236addceb52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:35:54.984311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.977106 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-metrics-certs" pod "21e851e1-0779-48c0-9e4b-2236addceb52" (UID: "21e851e1-0779-48c0-9e4b-2236addceb52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:54.984311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.983094 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e851e1-0779-48c0-9e4b-2236addceb52-kube-api-access-dmdjs" pod "21e851e1-0779-48c0-9e4b-2236addceb52" (UID: "21e851e1-0779-48c0-9e4b-2236addceb52"). InnerVolumeSpecName "kube-api-access-dmdjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:35:54.984311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:54.983182 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-maas-api-tls" pod "21e851e1-0779-48c0-9e4b-2236addceb52" (UID: "21e851e1-0779-48c0-9e4b-2236addceb52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:35:55.130785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.100295 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:55.130785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.100345 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21e851e1-0779-48c0-9e4b-2236addceb52-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:55.130785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.100361 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21e851e1-0779-48c0-9e4b-2236addceb52-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:55.130785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.100376 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdjs\" (UniqueName: \"kubernetes.io/projected/21e851e1-0779-48c0-9e4b-2236addceb52-kube-api-access-dmdjs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:35:55.135943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.135872 4828 generic.go:356] "Generic (PLEG): container finished" podID="21e851e1-0779-48c0-9e4b-2236addceb52" containerID="83b7f91715d3995db6fd63d6b6578410638caf8d4c92ce8c95ba1636d89e67b7" exitCode=0 Oct 30 17:35:55.136460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.136430 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" event={"ID":"21e851e1-0779-48c0-9e4b-2236addceb52","Type":"ContainerDied","Data":"83b7f91715d3995db6fd63d6b6578410638caf8d4c92ce8c95ba1636d89e67b7"} Oct 30 17:35:55.136659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.136639 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" event={"ID":"21e851e1-0779-48c0-9e4b-2236addceb52","Type":"ContainerDied","Data":"0514b2927385d0beeee280381f20746988e6bfdd291efcbbf4eb8672a7d1f824"} Oct 30 17:35:55.136808 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.136765 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n" Oct 30 17:35:55.136932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.136916 4828 scope.go:122] "RemoveContainer" containerID="83b7f91715d3995db6fd63d6b6578410638caf8d4c92ce8c95ba1636d89e67b7" Oct 30 17:35:55.150727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.150702 4828 scope.go:122] "RemoveContainer" containerID="83b7f91715d3995db6fd63d6b6578410638caf8d4c92ce8c95ba1636d89e67b7" Oct 30 17:35:55.151086 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:35:55.151060 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83b7f91715d3995db6fd63d6b6578410638caf8d4c92ce8c95ba1636d89e67b7\": container with ID starting with 83b7f91715d3995db6fd63d6b6578410638caf8d4c92ce8c95ba1636d89e67b7 not found: ID does not exist" containerID="83b7f91715d3995db6fd63d6b6578410638caf8d4c92ce8c95ba1636d89e67b7" Oct 30 17:35:55.151153 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.151093 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83b7f91715d3995db6fd63d6b6578410638caf8d4c92ce8c95ba1636d89e67b7"} err="failed to get container status \"83b7f91715d3995db6fd63d6b6578410638caf8d4c92ce8c95ba1636d89e67b7\": rpc error: code = NotFound desc = could not find container \"83b7f91715d3995db6fd63d6b6578410638caf8d4c92ce8c95ba1636d89e67b7\": container with ID starting with 83b7f91715d3995db6fd63d6b6578410638caf8d4c92ce8c95ba1636d89e67b7 not found: ID does not exist" Oct 30 17:35:55.167731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.167703 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 17:35:55.167895 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.167747 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 17:35:55.222093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.222062 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n"] Oct 30 17:35:55.226618 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.226585 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a43843-67588b788b-2pk5n"] Oct 30 17:35:55.301108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.300832 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:55.512483 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.512440 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:55.706799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.706757 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:55.862974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:55.855339 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:56.014754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:56.014704 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:56.168563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:56.168468 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 17:35:56.168747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:56.168553 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 17:35:56.314658 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:56.314619 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:56.508908 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:56.508877 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e851e1-0779-48c0-9e4b-2236addceb52" path="/var/lib/kubelet/pods/21e851e1-0779-48c0-9e4b-2236addceb52/volumes" Oct 30 17:35:57.168595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:57.168560 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 17:35:57.169143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:57.168617 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 17:35:57.978375 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:57.978322 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:35:58.168580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:58.168546 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 17:35:58.168773 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:58.168598 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 17:35:59.173690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:59.173653 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:35:59.174182 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:59.174083 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:35:59.175007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:59.174987 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:35:59.762655 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:35:59.762619 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:00.155415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:00.155326 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:00.793580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:00.793546 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:01.428922 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:01.425757 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:01.727850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:01.727753 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:01.880691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:01.880653 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:03.138708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:03.138667 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:03.314468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:03.314428 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:03.657305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:03.657272 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5"] Oct 30 17:36:03.657521 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:03.657485 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" podUID="16c8de1e-84a0-4f5c-9997-3393f2de5540" containerName="istio-proxy" containerID="cri-o://e967a115e07f093f344937831efe1fc580188b19f684c8d984e5a9939aa0fd2f" gracePeriod=30 Oct 30 17:36:03.667583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:03.667551 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:04.192359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:04.192319 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:04.540818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:04.540781 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:04.699316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:04.699282 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:05.711923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:05.711749 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:06.463740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:06.463694 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:06.618750 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:06.618713 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:07.916590 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:07.916545 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:36:07.929430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:07.929379 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:08.978166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:08.978128 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:36:09.105819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.105780 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-data\") pod \"16c8de1e-84a0-4f5c-9997-3393f2de5540\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " Oct 30 17:36:09.106143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.105853 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-podinfo\") pod \"16c8de1e-84a0-4f5c-9997-3393f2de5540\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " Oct 30 17:36:09.106143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.105953 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16c8de1e-84a0-4f5c-9997-3393f2de5540-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c8de1e-84a0-4f5c-9997-3393f2de5540-istiod-ca-cert\") pod \"16c8de1e-84a0-4f5c-9997-3393f2de5540\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " Oct 30 17:36:09.106143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.106015 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-kube-api-access-xl67s\" (UniqueName: \"kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-kube-api-access-xl67s\") pod \"16c8de1e-84a0-4f5c-9997-3393f2de5540\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " Oct 30 17:36:09.106143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.106127 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-certs\") pod \"16c8de1e-84a0-4f5c-9997-3393f2de5540\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " Oct 30 17:36:09.106384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.106161 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-data" pod "16c8de1e-84a0-4f5c-9997-3393f2de5540" (UID: "16c8de1e-84a0-4f5c-9997-3393f2de5540"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:09.106384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.106221 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-credential-socket\") pod \"16c8de1e-84a0-4f5c-9997-3393f2de5540\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " Oct 30 17:36:09.106384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.106250 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-envoy\") pod \"16c8de1e-84a0-4f5c-9997-3393f2de5540\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " Oct 30 17:36:09.106384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.106291 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-socket\") pod \"16c8de1e-84a0-4f5c-9997-3393f2de5540\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " Oct 30 17:36:09.106384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.106372 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-token\" (UniqueName: \"kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-token\") pod \"16c8de1e-84a0-4f5c-9997-3393f2de5540\" (UID: \"16c8de1e-84a0-4f5c-9997-3393f2de5540\") " Oct 30 17:36:09.106707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.106532 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-certs" pod "16c8de1e-84a0-4f5c-9997-3393f2de5540" (UID: "16c8de1e-84a0-4f5c-9997-3393f2de5540"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:09.106707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.106556 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c8de1e-84a0-4f5c-9997-3393f2de5540-istiod-ca-cert" pod "16c8de1e-84a0-4f5c-9997-3393f2de5540" (UID: "16c8de1e-84a0-4f5c-9997-3393f2de5540"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:09.106707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.106561 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-credential-socket" pod "16c8de1e-84a0-4f5c-9997-3393f2de5540" (UID: "16c8de1e-84a0-4f5c-9997-3393f2de5540"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:09.106889 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.106759 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-socket" pod "16c8de1e-84a0-4f5c-9997-3393f2de5540" (UID: "16c8de1e-84a0-4f5c-9997-3393f2de5540"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:09.107363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.107042 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16c8de1e-84a0-4f5c-9997-3393f2de5540-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:09.107363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.107061 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:09.107363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.107076 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:09.107363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.107091 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:09.107363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.107105 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:09.109938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.109865 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-kube-api-access-xl67s" pod "16c8de1e-84a0-4f5c-9997-3393f2de5540" (UID: "16c8de1e-84a0-4f5c-9997-3393f2de5540"). InnerVolumeSpecName "kube-api-access-xl67s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:09.111388 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.111271 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-envoy" pod "16c8de1e-84a0-4f5c-9997-3393f2de5540" (UID: "16c8de1e-84a0-4f5c-9997-3393f2de5540"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:09.112104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.112070 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-podinfo" pod "16c8de1e-84a0-4f5c-9997-3393f2de5540" (UID: "16c8de1e-84a0-4f5c-9997-3393f2de5540"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:36:09.112341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.112322 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-token" pod "16c8de1e-84a0-4f5c-9997-3393f2de5540" (UID: "16c8de1e-84a0-4f5c-9997-3393f2de5540"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:09.208052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.207950 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:09.208052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.207981 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:09.208052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.207990 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16c8de1e-84a0-4f5c-9997-3393f2de5540-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:09.208052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.208000 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl67s\" (UniqueName: \"kubernetes.io/projected/16c8de1e-84a0-4f5c-9997-3393f2de5540-kube-api-access-xl67s\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:09.232026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.231986 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:09.252635 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.252597 4828 generic.go:356] "Generic (PLEG): container finished" podID="16c8de1e-84a0-4f5c-9997-3393f2de5540" containerID="e967a115e07f093f344937831efe1fc580188b19f684c8d984e5a9939aa0fd2f" exitCode=0 Oct 30 17:36:09.252832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.252673 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" Oct 30 17:36:09.252832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.252685 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" event={"ID":"16c8de1e-84a0-4f5c-9997-3393f2de5540","Type":"ContainerDied","Data":"e967a115e07f093f344937831efe1fc580188b19f684c8d984e5a9939aa0fd2f"} Oct 30 17:36:09.252832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.252769 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5" event={"ID":"16c8de1e-84a0-4f5c-9997-3393f2de5540","Type":"ContainerDied","Data":"437ad09eac444e0b60b0131a08ee474f444d2dee60227bb4e8385f30645d6f2f"} Oct 30 17:36:09.252832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.252793 4828 scope.go:122] "RemoveContainer" containerID="e967a115e07f093f344937831efe1fc580188b19f684c8d984e5a9939aa0fd2f" Oct 30 17:36:09.266463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.266442 4828 scope.go:122] "RemoveContainer" containerID="e967a115e07f093f344937831efe1fc580188b19f684c8d984e5a9939aa0fd2f" Oct 30 17:36:09.266768 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:09.266746 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e967a115e07f093f344937831efe1fc580188b19f684c8d984e5a9939aa0fd2f\": container with ID starting with e967a115e07f093f344937831efe1fc580188b19f684c8d984e5a9939aa0fd2f not found: ID does not exist" containerID="e967a115e07f093f344937831efe1fc580188b19f684c8d984e5a9939aa0fd2f" Oct 30 17:36:09.266837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.266784 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e967a115e07f093f344937831efe1fc580188b19f684c8d984e5a9939aa0fd2f"} err="failed to get container status \"e967a115e07f093f344937831efe1fc580188b19f684c8d984e5a9939aa0fd2f\": rpc error: code = NotFound desc = could not find container \"e967a115e07f093f344937831efe1fc580188b19f684c8d984e5a9939aa0fd2f\": container with ID starting with e967a115e07f093f344937831efe1fc580188b19f684c8d984e5a9939aa0fd2f not found: ID does not exist" Oct 30 17:36:09.311956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.311920 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5"] Oct 30 17:36:09.323526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.323492 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a43843-openshift-default-665d97c97-p6cl5"] Oct 30 17:36:09.659637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:09.659597 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:10.494440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:10.494391 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c8de1e-84a0-4f5c-9997-3393f2de5540" path="/var/lib/kubelet/pods/16c8de1e-84a0-4f5c-9997-3393f2de5540/volumes" Oct 30 17:36:11.016241 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:11.016203 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:11.578411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:11.578380 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:11.747302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:11.747265 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:12.236700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:12.236668 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:13.336872 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:13.336835 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:14.522563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:14.522527 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js"] Oct 30 17:36:14.818758 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:14.818674 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:14.831513 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:14.831473 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6"] Oct 30 17:36:14.832953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:14.832918 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" podUID="e16084cf-b957-48f1-8c87-cf83e016da52" containerName="maas-api" containerID="cri-o://aac5f7d87c02c3ff238c36a4d222b8150974191803b4486db4d81be90a0324eb" gracePeriod=30 Oct 30 17:36:14.999924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:14.993306 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:15.154402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.154179 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:15.347505 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.347262 4828 generic.go:356] "Generic (PLEG): container finished" podID="e16084cf-b957-48f1-8c87-cf83e016da52" containerID="aac5f7d87c02c3ff238c36a4d222b8150974191803b4486db4d81be90a0324eb" exitCode=0 Oct 30 17:36:15.347505 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.347431 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" event={"ID":"e16084cf-b957-48f1-8c87-cf83e016da52","Type":"ContainerDied","Data":"aac5f7d87c02c3ff238c36a4d222b8150974191803b4486db4d81be90a0324eb"} Oct 30 17:36:15.542923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.540007 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:36:15.594980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.576814 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:15.641826 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.641789 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16084cf-b957-48f1-8c87-cf83e016da52-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16084cf-b957-48f1-8c87-cf83e016da52-openshift-service-ca\") pod \"e16084cf-b957-48f1-8c87-cf83e016da52\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " Oct 30 17:36:15.642050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.641840 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16084cf-b957-48f1-8c87-cf83e016da52-kube-api-access-5wbh2\" (UniqueName: \"kubernetes.io/projected/e16084cf-b957-48f1-8c87-cf83e016da52-kube-api-access-5wbh2\") pod \"e16084cf-b957-48f1-8c87-cf83e016da52\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " Oct 30 17:36:15.642050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.641889 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-metrics-certs\") pod \"e16084cf-b957-48f1-8c87-cf83e016da52\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " Oct 30 17:36:15.642050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.642011 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-maas-api-tls\") pod \"e16084cf-b957-48f1-8c87-cf83e016da52\" (UID: \"e16084cf-b957-48f1-8c87-cf83e016da52\") " Oct 30 17:36:15.643256 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.643213 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16084cf-b957-48f1-8c87-cf83e016da52-openshift-service-ca" pod "e16084cf-b957-48f1-8c87-cf83e016da52" (UID: "e16084cf-b957-48f1-8c87-cf83e016da52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:15.645294 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.645262 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-maas-api-tls" pod "e16084cf-b957-48f1-8c87-cf83e016da52" (UID: "e16084cf-b957-48f1-8c87-cf83e016da52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:15.652617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.652249 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16084cf-b957-48f1-8c87-cf83e016da52-kube-api-access-5wbh2" pod "e16084cf-b957-48f1-8c87-cf83e016da52" (UID: "e16084cf-b957-48f1-8c87-cf83e016da52"). InnerVolumeSpecName "kube-api-access-5wbh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:15.652854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.652832 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-metrics-certs" pod "e16084cf-b957-48f1-8c87-cf83e016da52" (UID: "e16084cf-b957-48f1-8c87-cf83e016da52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:15.735996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.735724 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:15.750059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.744151 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16084cf-b957-48f1-8c87-cf83e016da52-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:15.750376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.750336 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wbh2\" (UniqueName: \"kubernetes.io/projected/e16084cf-b957-48f1-8c87-cf83e016da52-kube-api-access-5wbh2\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:15.750516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.750503 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:15.750624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.750614 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16084cf-b957-48f1-8c87-cf83e016da52-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:15.916923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:15.916502 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:16.287924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:16.287636 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:16.357711 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:16.357674 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" Oct 30 17:36:16.357940 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:16.357708 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6" event={"ID":"e16084cf-b957-48f1-8c87-cf83e016da52","Type":"ContainerDied","Data":"cd6b77c30d6621be258111002336299e6d2184e3060867a2d2208ee91cc8a369"} Oct 30 17:36:16.357940 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:16.357783 4828 scope.go:122] "RemoveContainer" containerID="aac5f7d87c02c3ff238c36a4d222b8150974191803b4486db4d81be90a0324eb" Oct 30 17:36:16.418393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:16.418355 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6"] Oct 30 17:36:16.430021 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:16.429986 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-78c9c2-64cc6d48c8-gw4x6"] Oct 30 17:36:16.514258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:16.514223 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16084cf-b957-48f1-8c87-cf83e016da52" path="/var/lib/kubelet/pods/e16084cf-b957-48f1-8c87-cf83e016da52/volumes" Oct 30 17:36:17.351734 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:17.351695 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:17.513923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:17.513817 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:18.463371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:18.463333 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:19.328492 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:19.328456 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:19.505461 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:19.505401 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:19.522792 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:19.522754 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" podUID="b1dfb448-5107-40ae-a8a7-6766274ad9ce" containerName="payload-processing" containerID="cri-o://a7013b65c85d127e323cc94864ba063869fc56618176f3432a6353aea62e59f3" gracePeriod=25 Oct 30 17:36:19.632071 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:19.631968 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr"] Oct 30 17:36:19.632694 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:19.632664 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerName="istio-proxy" containerID="cri-o://39d53b47ea2b78af9e57bd31ca3d1e2da25bf9ebb0a05ee56716d908d40a2a5e" gracePeriod=30 Oct 30 17:36:19.705976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:19.695482 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:20.238629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:20.238590 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:20.429166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:20.429117 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:22.484925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:22.481345 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:23.584076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:23.584032 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:23.751554 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:23.751516 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:23.961425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:23.961297 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:24.179135 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:24.179085 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:36:24.179424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:24.179190 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:36:24.498733 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:24.498698 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:24.912056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:24.912030 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:36:25.005349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.005304 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istiod-ca-cert\") pod \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " Oct 30 17:36:25.005567 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.005368 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-socket\") pod \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " Oct 30 17:36:25.005567 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.005404 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-data\") pod \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " Oct 30 17:36:25.005567 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.005529 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-kube-api-access-9xcvs\" (UniqueName: \"kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-kube-api-access-9xcvs\") pod \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " Oct 30 17:36:25.005788 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.005664 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-certs\") pod \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " Oct 30 17:36:25.005788 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.005694 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-socket" pod "5f3f407f-bf79-4e39-8c17-c1e2a33eadae" (UID: "5f3f407f-bf79-4e39-8c17-c1e2a33eadae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:25.005788 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.005705 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-podinfo\") pod \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " Oct 30 17:36:25.005788 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.005754 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-envoy\") pod \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " Oct 30 17:36:25.006095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.005824 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-credential-socket\") pod \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " Oct 30 17:36:25.006095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.005966 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-token\" (UniqueName: \"kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-token\") pod \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\" (UID: \"5f3f407f-bf79-4e39-8c17-c1e2a33eadae\") " Oct 30 17:36:25.006095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.006066 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-certs" pod "5f3f407f-bf79-4e39-8c17-c1e2a33eadae" (UID: "5f3f407f-bf79-4e39-8c17-c1e2a33eadae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:25.006271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.006141 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-credential-socket" pod "5f3f407f-bf79-4e39-8c17-c1e2a33eadae" (UID: "5f3f407f-bf79-4e39-8c17-c1e2a33eadae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:25.006271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.006230 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-data" pod "5f3f407f-bf79-4e39-8c17-c1e2a33eadae" (UID: "5f3f407f-bf79-4e39-8c17-c1e2a33eadae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:25.006379 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.006263 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istiod-ca-cert" pod "5f3f407f-bf79-4e39-8c17-c1e2a33eadae" (UID: "5f3f407f-bf79-4e39-8c17-c1e2a33eadae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:25.006634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.006616 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.006718 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.006638 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.006718 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.006651 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.006718 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.006665 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.006718 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.006678 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.009247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.009219 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-envoy" pod "5f3f407f-bf79-4e39-8c17-c1e2a33eadae" (UID: "5f3f407f-bf79-4e39-8c17-c1e2a33eadae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:25.009342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.009250 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-podinfo" pod "5f3f407f-bf79-4e39-8c17-c1e2a33eadae" (UID: "5f3f407f-bf79-4e39-8c17-c1e2a33eadae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:36:25.009342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.009284 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-kube-api-access-9xcvs" pod "5f3f407f-bf79-4e39-8c17-c1e2a33eadae" (UID: "5f3f407f-bf79-4e39-8c17-c1e2a33eadae"). InnerVolumeSpecName "kube-api-access-9xcvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:25.010469 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.010450 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-token" pod "5f3f407f-bf79-4e39-8c17-c1e2a33eadae" (UID: "5f3f407f-bf79-4e39-8c17-c1e2a33eadae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:25.108480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.108386 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.108480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.108422 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.108480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.108432 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.108480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.108441 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xcvs\" (UniqueName: \"kubernetes.io/projected/5f3f407f-bf79-4e39-8c17-c1e2a33eadae-kube-api-access-9xcvs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:25.173266 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.173224 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m"] Oct 30 17:36:25.246752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.246714 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" podUID="b1dfb448-5107-40ae-a8a7-6766274ad9ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:25.317396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.317348 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:25.529564 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.529531 4828 generic.go:356] "Generic (PLEG): container finished" podID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerID="39d53b47ea2b78af9e57bd31ca3d1e2da25bf9ebb0a05ee56716d908d40a2a5e" exitCode=0 Oct 30 17:36:25.529780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.529587 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" Oct 30 17:36:25.529780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.529658 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" event={"ID":"5f3f407f-bf79-4e39-8c17-c1e2a33eadae","Type":"ContainerDied","Data":"39d53b47ea2b78af9e57bd31ca3d1e2da25bf9ebb0a05ee56716d908d40a2a5e"} Oct 30 17:36:25.529780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.529718 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr" event={"ID":"5f3f407f-bf79-4e39-8c17-c1e2a33eadae","Type":"ContainerDied","Data":"001102abcc95e038cb7249f31d610c6bb165eb6e9f1381eee58f976f396505f9"} Oct 30 17:36:25.529780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.529756 4828 scope.go:122] "RemoveContainer" containerID="39d53b47ea2b78af9e57bd31ca3d1e2da25bf9ebb0a05ee56716d908d40a2a5e" Oct 30 17:36:25.531799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.531775 4828 generic.go:356] "Generic (PLEG): container finished" podID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerID="ebcf70a9fc3860f0899fc2c6231c5abe4de1bf6de2371db824b856ce1a70a097" exitCode=0 Oct 30 17:36:25.531942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.531801 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" event={"ID":"c249be1c-0e0f-4380-917c-bb42dc4308c8","Type":"ContainerDied","Data":"ebcf70a9fc3860f0899fc2c6231c5abe4de1bf6de2371db824b856ce1a70a097"} Oct 30 17:36:25.532296 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.532275 4828 scope.go:122] "RemoveContainer" containerID="ebcf70a9fc3860f0899fc2c6231c5abe4de1bf6de2371db824b856ce1a70a097" Oct 30 17:36:25.533881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.533851 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:36:25.534035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.533939 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:36:25.534035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.533971 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:36:25.544112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.544087 4828 scope.go:122] "RemoveContainer" containerID="e2a2a6d091bbc13b550b409f779e3e3234b92a3827254279450b685a2e618cd0" Oct 30 17:36:25.561669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.561595 4828 scope.go:122] "RemoveContainer" containerID="39d53b47ea2b78af9e57bd31ca3d1e2da25bf9ebb0a05ee56716d908d40a2a5e" Oct 30 17:36:25.562101 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:25.562079 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d53b47ea2b78af9e57bd31ca3d1e2da25bf9ebb0a05ee56716d908d40a2a5e\": container with ID starting with 39d53b47ea2b78af9e57bd31ca3d1e2da25bf9ebb0a05ee56716d908d40a2a5e not found: ID does not exist" containerID="39d53b47ea2b78af9e57bd31ca3d1e2da25bf9ebb0a05ee56716d908d40a2a5e" Oct 30 17:36:25.562184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.562109 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d53b47ea2b78af9e57bd31ca3d1e2da25bf9ebb0a05ee56716d908d40a2a5e"} err="failed to get container status \"39d53b47ea2b78af9e57bd31ca3d1e2da25bf9ebb0a05ee56716d908d40a2a5e\": rpc error: code = NotFound desc = could not find container \"39d53b47ea2b78af9e57bd31ca3d1e2da25bf9ebb0a05ee56716d908d40a2a5e\": container with ID starting with 39d53b47ea2b78af9e57bd31ca3d1e2da25bf9ebb0a05ee56716d908d40a2a5e not found: ID does not exist" Oct 30 17:36:25.562184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.562126 4828 scope.go:122] "RemoveContainer" containerID="e2a2a6d091bbc13b550b409f779e3e3234b92a3827254279450b685a2e618cd0" Oct 30 17:36:25.562421 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:25.562395 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a2a6d091bbc13b550b409f779e3e3234b92a3827254279450b685a2e618cd0\": container with ID starting with e2a2a6d091bbc13b550b409f779e3e3234b92a3827254279450b685a2e618cd0 not found: ID does not exist" containerID="e2a2a6d091bbc13b550b409f779e3e3234b92a3827254279450b685a2e618cd0" Oct 30 17:36:25.562479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.562434 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a2a6d091bbc13b550b409f779e3e3234b92a3827254279450b685a2e618cd0"} err="failed to get container status \"e2a2a6d091bbc13b550b409f779e3e3234b92a3827254279450b685a2e618cd0\": rpc error: code = NotFound desc = could not find container \"e2a2a6d091bbc13b550b409f779e3e3234b92a3827254279450b685a2e618cd0\": container with ID starting with e2a2a6d091bbc13b550b409f779e3e3234b92a3827254279450b685a2e618cd0 not found: ID does not exist" Oct 30 17:36:25.637611 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.637577 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr"] Oct 30 17:36:25.647466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.647439 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d4f99-openshift-default-67854bc4d5-bhdgr"] Oct 30 17:36:25.961084 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.961001 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w"] Oct 30 17:36:25.961527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:25.961242 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" podUID="96c7bccf-7a2e-464e-984f-8c6dc196d1c5" containerName="maas-api" containerID="cri-o://886e826456b809ac9f85003b0ba5c4afb53a684d3f50f7634b72b086d43416d9" gracePeriod=30 Oct 30 17:36:26.103600 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.103554 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:26.357829 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.357804 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:36:26.428736 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.427111 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-maas-api-tls\") pod \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " Oct 30 17:36:26.428736 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.427202 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-kube-api-access-r7tvn\" (UniqueName: \"kubernetes.io/projected/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-kube-api-access-r7tvn\") pod \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " Oct 30 17:36:26.428736 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.427269 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-metrics-certs\") pod \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " Oct 30 17:36:26.428736 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.427355 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-openshift-service-ca\") pod \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\" (UID: \"96c7bccf-7a2e-464e-984f-8c6dc196d1c5\") " Oct 30 17:36:26.429652 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.429315 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-openshift-service-ca" pod "96c7bccf-7a2e-464e-984f-8c6dc196d1c5" (UID: "96c7bccf-7a2e-464e-984f-8c6dc196d1c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:26.440116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.440046 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-maas-api-tls" pod "96c7bccf-7a2e-464e-984f-8c6dc196d1c5" (UID: "96c7bccf-7a2e-464e-984f-8c6dc196d1c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:26.469629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.469581 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb"] Oct 30 17:36:26.472281 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.472216 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-kube-api-access-r7tvn" pod "96c7bccf-7a2e-464e-984f-8c6dc196d1c5" (UID: "96c7bccf-7a2e-464e-984f-8c6dc196d1c5"). InnerVolumeSpecName "kube-api-access-r7tvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:26.472868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.472833 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-metrics-certs" pod "96c7bccf-7a2e-464e-984f-8c6dc196d1c5" (UID: "96c7bccf-7a2e-464e-984f-8c6dc196d1c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:26.529408 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.529345 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:26.529670 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.529652 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:26.529782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.529768 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7tvn\" (UniqueName: \"kubernetes.io/projected/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-kube-api-access-r7tvn\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:26.529875 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.529864 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96c7bccf-7a2e-464e-984f-8c6dc196d1c5-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:26.629013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.626671 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" path="/var/lib/kubelet/pods/5f3f407f-bf79-4e39-8c17-c1e2a33eadae/volumes" Oct 30 17:36:26.629013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.627593 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk"] Oct 30 17:36:26.631677 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.631649 4828 generic.go:356] "Generic (PLEG): container finished" podID="96c7bccf-7a2e-464e-984f-8c6dc196d1c5" containerID="886e826456b809ac9f85003b0ba5c4afb53a684d3f50f7634b72b086d43416d9" exitCode=0 Oct 30 17:36:26.632021 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.632001 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" event={"ID":"96c7bccf-7a2e-464e-984f-8c6dc196d1c5","Type":"ContainerDied","Data":"886e826456b809ac9f85003b0ba5c4afb53a684d3f50f7634b72b086d43416d9"} Oct 30 17:36:26.632178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.632162 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" event={"ID":"96c7bccf-7a2e-464e-984f-8c6dc196d1c5","Type":"ContainerDied","Data":"6f6b5f956cd8e70b1cb822f925ed59c9e0afb76df1439bafe0ecbea5c6876a53"} Oct 30 17:36:26.632293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.632281 4828 scope.go:122] "RemoveContainer" containerID="886e826456b809ac9f85003b0ba5c4afb53a684d3f50f7634b72b086d43416d9" Oct 30 17:36:26.632498 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.632485 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w" Oct 30 17:36:26.656598 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.656557 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" event={"ID":"c249be1c-0e0f-4380-917c-bb42dc4308c8","Type":"ContainerStarted","Data":"0881742fccc5d7c0fe444d64762aa399c1e52ff51c3a8909094225d1c935ed4e"} Oct 30 17:36:26.659293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.659274 4828 scope.go:122] "RemoveContainer" containerID="886e826456b809ac9f85003b0ba5c4afb53a684d3f50f7634b72b086d43416d9" Oct 30 17:36:26.662046 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:26.662020 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886e826456b809ac9f85003b0ba5c4afb53a684d3f50f7634b72b086d43416d9\": container with ID starting with 886e826456b809ac9f85003b0ba5c4afb53a684d3f50f7634b72b086d43416d9 not found: ID does not exist" containerID="886e826456b809ac9f85003b0ba5c4afb53a684d3f50f7634b72b086d43416d9" Oct 30 17:36:26.662226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.662196 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886e826456b809ac9f85003b0ba5c4afb53a684d3f50f7634b72b086d43416d9"} err="failed to get container status \"886e826456b809ac9f85003b0ba5c4afb53a684d3f50f7634b72b086d43416d9\": rpc error: code = NotFound desc = could not find container \"886e826456b809ac9f85003b0ba5c4afb53a684d3f50f7634b72b086d43416d9\": container with ID starting with 886e826456b809ac9f85003b0ba5c4afb53a684d3f50f7634b72b086d43416d9 not found: ID does not exist" Oct 30 17:36:26.720920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.717512 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:26.829926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.795603 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w"] Oct 30 17:36:26.850755 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.850670 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-db8417-5c787d5cb5-pqw6w"] Oct 30 17:36:26.896008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:26.895971 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:27.047484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:27.047438 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:27.204026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:27.203592 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:27.204701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:27.204677 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:36:27.205022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:27.205002 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 17:36:27.205172 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:27.205151 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 17:36:27.355753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:27.355703 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:27.760662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:27.760624 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:28.209982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:28.203296 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 17:36:28.209982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:28.203367 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 17:36:28.508995 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:28.508961 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c7bccf-7a2e-464e-984f-8c6dc196d1c5" path="/var/lib/kubelet/pods/96c7bccf-7a2e-464e-984f-8c6dc196d1c5/volumes" Oct 30 17:36:29.121852 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:29.119965 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:29.201418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:29.201386 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 17:36:29.201663 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:29.201640 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 17:36:30.201437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:30.201391 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 17:36:30.202013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:30.201464 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 17:36:30.710747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:30.710460 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:30.896925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:30.896086 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:31.168600 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:31.168568 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:31.204334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:31.204296 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 17:36:31.204973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:31.204894 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 17:36:31.474074 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:31.473968 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" podUID="c2b37307-2b2b-4e37-87f1-a1e898d17e3d" containerName="payload-processing" containerID="cri-o://31ef76a3f7f17c16b5de96d52bc3d197b0ac5dd35f76c6ea99005138bd343ca9" gracePeriod=25 Oct 30 17:36:31.628556 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:31.628481 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" podUID="8b51c1d9-8eb1-479a-9321-ee204c391050" containerName="payload-pre-processing" containerID="cri-o://bcaa21b66dfd173e3fdaa6b7486ac710cb28201d7636534f65fa39a747088e62" gracePeriod=25 Oct 30 17:36:31.746801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:31.746627 4828 generic.go:356] "Generic (PLEG): container finished" podID="c2b37307-2b2b-4e37-87f1-a1e898d17e3d" containerID="31ef76a3f7f17c16b5de96d52bc3d197b0ac5dd35f76c6ea99005138bd343ca9" exitCode=0 Oct 30 17:36:31.746801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:31.746664 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" event={"ID":"c2b37307-2b2b-4e37-87f1-a1e898d17e3d","Type":"ContainerDied","Data":"31ef76a3f7f17c16b5de96d52bc3d197b0ac5dd35f76c6ea99005138bd343ca9"} Oct 30 17:36:31.863891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:31.863863 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" Oct 30 17:36:31.994923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:31.994371 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-kube-api-access-266dd\" (UniqueName: \"kubernetes.io/projected/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-kube-api-access-266dd\") pod \"c2b37307-2b2b-4e37-87f1-a1e898d17e3d\" (UID: \"c2b37307-2b2b-4e37-87f1-a1e898d17e3d\") " Oct 30 17:36:31.994923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:31.994565 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-plugins-config-volume\") pod \"c2b37307-2b2b-4e37-87f1-a1e898d17e3d\" (UID: \"c2b37307-2b2b-4e37-87f1-a1e898d17e3d\") " Oct 30 17:36:32.008994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:31.996625 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-plugins-config-volume" pod "c2b37307-2b2b-4e37-87f1-a1e898d17e3d" (UID: "c2b37307-2b2b-4e37-87f1-a1e898d17e3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:32.008994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.002522 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:32.014073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.013650 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-kube-api-access-266dd" pod "c2b37307-2b2b-4e37-87f1-a1e898d17e3d" (UID: "c2b37307-2b2b-4e37-87f1-a1e898d17e3d"). InnerVolumeSpecName "kube-api-access-266dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:32.038674 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.038636 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" Oct 30 17:36:32.106014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.104465 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b51c1d9-8eb1-479a-9321-ee204c391050-kube-api-access-h79rm\" (UniqueName: \"kubernetes.io/projected/8b51c1d9-8eb1-479a-9321-ee204c391050-kube-api-access-h79rm\") pod \"8b51c1d9-8eb1-479a-9321-ee204c391050\" (UID: \"8b51c1d9-8eb1-479a-9321-ee204c391050\") " Oct 30 17:36:32.106014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.104675 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b51c1d9-8eb1-479a-9321-ee204c391050-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b51c1d9-8eb1-479a-9321-ee204c391050-plugins-config-volume\") pod \"8b51c1d9-8eb1-479a-9321-ee204c391050\" (UID: \"8b51c1d9-8eb1-479a-9321-ee204c391050\") " Oct 30 17:36:32.106014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.105232 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-266dd\" (UniqueName: \"kubernetes.io/projected/c2b37307-2b2b-4e37-87f1-a1e898d17e3d-kube-api-access-266dd\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:32.106014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.105647 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b51c1d9-8eb1-479a-9321-ee204c391050-plugins-config-volume" pod "8b51c1d9-8eb1-479a-9321-ee204c391050" (UID: "8b51c1d9-8eb1-479a-9321-ee204c391050"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:32.108796 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.108730 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b51c1d9-8eb1-479a-9321-ee204c391050-kube-api-access-h79rm" pod "8b51c1d9-8eb1-479a-9321-ee204c391050" (UID: "8b51c1d9-8eb1-479a-9321-ee204c391050"). InnerVolumeSpecName "kube-api-access-h79rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:32.201049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.201002 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 17:36:32.201199 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.201076 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 17:36:32.207100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.207044 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b51c1d9-8eb1-479a-9321-ee204c391050-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:32.207100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.207085 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h79rm\" (UniqueName: \"kubernetes.io/projected/8b51c1d9-8eb1-479a-9321-ee204c391050-kube-api-access-h79rm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:32.755316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.755221 4828 generic.go:356] "Generic (PLEG): container finished" podID="8b51c1d9-8eb1-479a-9321-ee204c391050" containerID="bcaa21b66dfd173e3fdaa6b7486ac710cb28201d7636534f65fa39a747088e62" exitCode=0 Oct 30 17:36:32.755316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.755300 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" Oct 30 17:36:32.755580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.755313 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" event={"ID":"8b51c1d9-8eb1-479a-9321-ee204c391050","Type":"ContainerDied","Data":"bcaa21b66dfd173e3fdaa6b7486ac710cb28201d7636534f65fa39a747088e62"} Oct 30 17:36:32.755580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.755382 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk" event={"ID":"8b51c1d9-8eb1-479a-9321-ee204c391050","Type":"ContainerDied","Data":"973e5866e50bf50916abb735c42f269ac2a2b7dadcc5a40bd38c063e757358b2"} Oct 30 17:36:32.755580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.755413 4828 scope.go:122] "RemoveContainer" containerID="bcaa21b66dfd173e3fdaa6b7486ac710cb28201d7636534f65fa39a747088e62" Oct 30 17:36:32.757016 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.756976 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" event={"ID":"c2b37307-2b2b-4e37-87f1-a1e898d17e3d","Type":"ContainerDied","Data":"fd6d27e523efccbf0ccc778e96980d33cd292274c63d6c5b2c7c19efea1f926e"} Oct 30 17:36:32.757016 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.757004 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb" Oct 30 17:36:32.766810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.766790 4828 scope.go:122] "RemoveContainer" containerID="bcaa21b66dfd173e3fdaa6b7486ac710cb28201d7636534f65fa39a747088e62" Oct 30 17:36:32.767102 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:32.767082 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcaa21b66dfd173e3fdaa6b7486ac710cb28201d7636534f65fa39a747088e62\": container with ID starting with bcaa21b66dfd173e3fdaa6b7486ac710cb28201d7636534f65fa39a747088e62 not found: ID does not exist" containerID="bcaa21b66dfd173e3fdaa6b7486ac710cb28201d7636534f65fa39a747088e62" Oct 30 17:36:32.767182 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.767116 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcaa21b66dfd173e3fdaa6b7486ac710cb28201d7636534f65fa39a747088e62"} err="failed to get container status \"bcaa21b66dfd173e3fdaa6b7486ac710cb28201d7636534f65fa39a747088e62\": rpc error: code = NotFound desc = could not find container \"bcaa21b66dfd173e3fdaa6b7486ac710cb28201d7636534f65fa39a747088e62\": container with ID starting with bcaa21b66dfd173e3fdaa6b7486ac710cb28201d7636534f65fa39a747088e62 not found: ID does not exist" Oct 30 17:36:32.767182 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.767139 4828 scope.go:122] "RemoveContainer" containerID="31ef76a3f7f17c16b5de96d52bc3d197b0ac5dd35f76c6ea99005138bd343ca9" Oct 30 17:36:32.812447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.812411 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb"] Oct 30 17:36:32.825247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.825218 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-031223-64cf779d96-n8lcb"] Oct 30 17:36:32.852432 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.852404 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk"] Oct 30 17:36:32.860060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.860029 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-031223-56f78d5cc9-k47zk"] Oct 30 17:36:32.923428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:32.923385 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:33.207707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:33.207676 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:36:33.208365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:33.208336 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:36:33.209344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:33.209318 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:36:34.086430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.086392 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:34.494007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.493892 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b51c1d9-8eb1-479a-9321-ee204c391050" path="/var/lib/kubelet/pods/8b51c1d9-8eb1-479a-9321-ee204c391050/volumes" Oct 30 17:36:34.494494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.494399 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b37307-2b2b-4e37-87f1-a1e898d17e3d" path="/var/lib/kubelet/pods/c2b37307-2b2b-4e37-87f1-a1e898d17e3d/volumes" Oct 30 17:36:34.537610 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.537574 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:34.632935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.624739 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt"] Oct 30 17:36:34.632935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.625781 4828 cpu_manager.go:414] "containerMap: removing container" podUID="7ab756a2-144f-4411-85db-dcb56009529f" containerName="istio-proxy" Oct 30 17:36:34.632935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.625802 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab756a2-144f-4411-85db-dcb56009529f" containerName="istio-proxy" Oct 30 17:36:34.632935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.625832 4828 cpu_manager.go:414] "containerMap: removing container" podUID="e84d9cf0-b5f6-4e24-b189-4e80e341e9f9" containerName="payload-pre-processing" Oct 30 17:36:34.632935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.625842 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84d9cf0-b5f6-4e24-b189-4e80e341e9f9" containerName="payload-pre-processing" Oct 30 17:36:34.632935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.625855 4828 cpu_manager.go:414] "containerMap: removing container" podUID="18f8f535-a840-4e43-8724-a408f694bf6c" containerName="payload-pre-processing" Oct 30 17:36:34.632935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.625863 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18f8f535-a840-4e43-8724-a408f694bf6c" containerName="payload-pre-processing" Oct 30 17:36:34.632935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.625877 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c2b37307-2b2b-4e37-87f1-a1e898d17e3d" containerName="payload-processing" Oct 30 17:36:34.632935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.625884 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b37307-2b2b-4e37-87f1-a1e898d17e3d" containerName="payload-processing" Oct 30 17:36:34.634184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.634156 4828 cpu_manager.go:414] "containerMap: removing container" podUID="e16084cf-b957-48f1-8c87-cf83e016da52" containerName="maas-api" Oct 30 17:36:34.634349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.634333 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16084cf-b957-48f1-8c87-cf83e016da52" containerName="maas-api" Oct 30 17:36:34.634473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.634460 4828 cpu_manager.go:414] "containerMap: removing container" podUID="16c8de1e-84a0-4f5c-9997-3393f2de5540" containerName="istio-proxy" Oct 30 17:36:34.634570 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.634556 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c8de1e-84a0-4f5c-9997-3393f2de5540" containerName="istio-proxy" Oct 30 17:36:34.634664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.634653 4828 cpu_manager.go:414] "containerMap: removing container" podUID="32a43570-25ea-4738-9109-85aaf132547d" containerName="payload-processing" Oct 30 17:36:34.634737 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.634727 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a43570-25ea-4738-9109-85aaf132547d" containerName="payload-processing" Oct 30 17:36:34.634816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.634807 4828 cpu_manager.go:414] "containerMap: removing container" podUID="21e851e1-0779-48c0-9e4b-2236addceb52" containerName="maas-api" Oct 30 17:36:34.634883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.634875 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e851e1-0779-48c0-9e4b-2236addceb52" containerName="maas-api" Oct 30 17:36:34.634979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.634968 4828 cpu_manager.go:414] "containerMap: removing container" podUID="3d9c5027-e57b-4b81-bdaf-3a8910099773" containerName="payload-processing" Oct 30 17:36:34.635044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.635035 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d9c5027-e57b-4b81-bdaf-3a8910099773" containerName="payload-processing" Oct 30 17:36:34.635105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.635098 4828 cpu_manager.go:414] "containerMap: removing container" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerName="istio-proxy" Oct 30 17:36:34.635174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.635165 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerName="istio-proxy" Oct 30 17:36:34.635335 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.635323 4828 cpu_manager.go:414] "containerMap: removing container" podUID="d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9" containerName="maas-api" Oct 30 17:36:34.635411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.635402 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9" containerName="maas-api" Oct 30 17:36:34.635482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.635473 4828 cpu_manager.go:414] "containerMap: removing container" podUID="83781a17-2754-43ad-9832-6bf1700b0eeb" containerName="maas-api" Oct 30 17:36:34.635548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.635539 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83781a17-2754-43ad-9832-6bf1700b0eeb" containerName="maas-api" Oct 30 17:36:34.635624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.635615 4828 cpu_manager.go:414] "containerMap: removing container" podUID="8b51c1d9-8eb1-479a-9321-ee204c391050" containerName="payload-pre-processing" Oct 30 17:36:34.635691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.635681 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b51c1d9-8eb1-479a-9321-ee204c391050" containerName="payload-pre-processing" Oct 30 17:36:34.635761 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.635751 4828 cpu_manager.go:414] "containerMap: removing container" podUID="96c7bccf-7a2e-464e-984f-8c6dc196d1c5" containerName="maas-api" Oct 30 17:36:34.635829 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.635820 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c7bccf-7a2e-464e-984f-8c6dc196d1c5" containerName="maas-api" Oct 30 17:36:34.635913 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.635889 4828 cpu_manager.go:414] "containerMap: removing container" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerName="istio-proxy" Oct 30 17:36:34.635994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.635984 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerName="istio-proxy" Oct 30 17:36:34.636065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.636057 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e" containerName="maas-api" Oct 30 17:36:34.636125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.636115 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e" containerName="maas-api" Oct 30 17:36:34.636396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.636382 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84d9cf0-b5f6-4e24-b189-4e80e341e9f9" containerName="payload-pre-processing" Oct 30 17:36:34.636492 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.636482 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="18f8f535-a840-4e43-8724-a408f694bf6c" containerName="payload-pre-processing" Oct 30 17:36:34.636562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.636554 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b37307-2b2b-4e37-87f1-a1e898d17e3d" containerName="payload-processing" Oct 30 17:36:34.636633 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.636625 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16084cf-b957-48f1-8c87-cf83e016da52" containerName="maas-api" Oct 30 17:36:34.636699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.636691 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c8de1e-84a0-4f5c-9997-3393f2de5540" containerName="istio-proxy" Oct 30 17:36:34.636770 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.636759 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a43570-25ea-4738-9109-85aaf132547d" containerName="payload-processing" Oct 30 17:36:34.636849 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.636840 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e851e1-0779-48c0-9e4b-2236addceb52" containerName="maas-api" Oct 30 17:36:34.636951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.636940 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d9c5027-e57b-4b81-bdaf-3a8910099773" containerName="payload-processing" Oct 30 17:36:34.637051 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.637040 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerName="istio-proxy" Oct 30 17:36:34.637120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.637113 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b8b7cd-dff7-48cd-a3ab-bfd3dfad5ee9" containerName="maas-api" Oct 30 17:36:34.637286 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.637270 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="83781a17-2754-43ad-9832-6bf1700b0eeb" containerName="maas-api" Oct 30 17:36:34.637414 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.637395 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b51c1d9-8eb1-479a-9321-ee204c391050" containerName="payload-pre-processing" Oct 30 17:36:34.637505 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.637494 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c7bccf-7a2e-464e-984f-8c6dc196d1c5" containerName="maas-api" Oct 30 17:36:34.637609 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.637594 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3f407f-bf79-4e39-8c17-c1e2a33eadae" containerName="istio-proxy" Oct 30 17:36:34.637680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.637670 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58a6f81-69f4-4bcd-bbf9-0bc3c93ab81e" containerName="maas-api" Oct 30 17:36:34.637776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.637765 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab756a2-144f-4411-85db-dcb56009529f" containerName="istio-proxy" Oct 30 17:36:34.646672 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.646645 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.665294 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.665260 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-054acd-openshift-default-dockercfg-nfkm6\"" Oct 30 17:36:34.722359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.722313 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:34.727919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.727873 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt"] Oct 30 17:36:34.754851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.754768 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-token\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.755141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.755112 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-envoy\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.755428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.755406 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-certs\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.755587 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.755568 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-socket\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.755755 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.755736 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slt7j\" (UniqueName: \"kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-kube-api-access-slt7j\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.755881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.755864 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-credential-socket\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.756035 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.756018 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-data\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.756201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.756181 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-podinfo\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.756333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.756316 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3660fa4-e79d-4624-a792-d60275ef4c43-istiod-ca-cert\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.858439 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.858407 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slt7j\" (UniqueName: \"kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-kube-api-access-slt7j\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.858643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.858481 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-credential-socket\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.858958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.858886 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-credential-socket\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.859138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.858988 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-data\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.859310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.859276 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-data\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.859751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.859549 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-podinfo\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.859949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.859927 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3660fa4-e79d-4624-a792-d60275ef4c43-istiod-ca-cert\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.860384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.860360 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-token\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.860491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.860411 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-envoy\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.861213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.860771 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-certs\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.861213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.860819 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-socket\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.861213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.861046 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3660fa4-e79d-4624-a792-d60275ef4c43-istiod-ca-cert\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.861537 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.861517 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-socket\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.862092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.862071 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-certs\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.864553 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.864525 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-envoy\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.866532 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.866507 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-podinfo\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.873662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.873640 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2"] Oct 30 17:36:34.874047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.874003 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" containerID="cri-o://e8b359a17219267c2d4c5c0c79203547bd088ca2ccd224f9264d8226ac735b85" gracePeriod=30 Oct 30 17:36:34.882066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.881939 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:34.886083 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.886031 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-token\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:34.902919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:34.891535 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slt7j\" (UniqueName: \"kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-kube-api-access-slt7j\") pod \"e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:35.028168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:35.027345 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:35.033587 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:35.033356 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:35.247811 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:35.247769 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" podUID="b1dfb448-5107-40ae-a8a7-6766274ad9ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:36:35.442924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:35.427266 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt"] Oct 30 17:36:35.464132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:35.443227 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:35.464132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:35.449165 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:36:35.464132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:35.449269 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:36:35.464132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:35.449316 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:36:35.792990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:35.792948 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" event={"ID":"a3660fa4-e79d-4624-a792-d60275ef4c43","Type":"ContainerStarted","Data":"503d48ce2722fd90e8109e1a7907d3b4bd73fc0451a7875047a1961b6b97b6bb"} Oct 30 17:36:35.793410 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:35.793022 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" event={"ID":"a3660fa4-e79d-4624-a792-d60275ef4c43","Type":"ContainerStarted","Data":"2d7dbcddabbd8d2fbeefaada34ad414db8be741866079e01cb558fdb663b7684"} Oct 30 17:36:35.845784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:35.845708 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" podStartSLOduration=1.845687149 podStartE2EDuration="1.845687149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:36:35.844103447 +0000 UTC m=+2973.489613626" watchObservedRunningTime="2026-10-30 17:36:35.845687149 +0000 UTC m=+2973.491197331" Oct 30 17:36:36.028162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:36.028118 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:36.034526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:36.034496 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:36.801304 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:36.801244 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:36.802353 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:36.802329 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:36:37.185278 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:37.185187 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:37.914318 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:37.914275 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:36:38.386732 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:38.386695 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:38.548831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:38.548794 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:39.094329 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.094286 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:39.151600 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.151549 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:36:39.151781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.151634 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:36:39.834018 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.833984 4828 generic.go:356] "Generic (PLEG): container finished" podID="b1dfb448-5107-40ae-a8a7-6766274ad9ce" containerID="a7013b65c85d127e323cc94864ba063869fc56618176f3432a6353aea62e59f3" exitCode=0 Oct 30 17:36:39.834157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.834039 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" event={"ID":"b1dfb448-5107-40ae-a8a7-6766274ad9ce","Type":"ContainerDied","Data":"a7013b65c85d127e323cc94864ba063869fc56618176f3432a6353aea62e59f3"} Oct 30 17:36:39.834157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.834088 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" event={"ID":"b1dfb448-5107-40ae-a8a7-6766274ad9ce","Type":"ContainerDied","Data":"7e2007462eb681df783c95062c485f45cf7dc9d15fc09df91e70917798edc1a4"} Oct 30 17:36:39.834157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.834108 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e2007462eb681df783c95062c485f45cf7dc9d15fc09df91e70917798edc1a4" Oct 30 17:36:39.835236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.835213 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" Oct 30 17:36:39.846324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.846290 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1dfb448-5107-40ae-a8a7-6766274ad9ce-kube-api-access-9b9f8\" (UniqueName: \"kubernetes.io/projected/b1dfb448-5107-40ae-a8a7-6766274ad9ce-kube-api-access-9b9f8\") pod \"b1dfb448-5107-40ae-a8a7-6766274ad9ce\" (UID: \"b1dfb448-5107-40ae-a8a7-6766274ad9ce\") " Oct 30 17:36:39.846443 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.846383 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1dfb448-5107-40ae-a8a7-6766274ad9ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1dfb448-5107-40ae-a8a7-6766274ad9ce-plugins-config-volume\") pod \"b1dfb448-5107-40ae-a8a7-6766274ad9ce\" (UID: \"b1dfb448-5107-40ae-a8a7-6766274ad9ce\") " Oct 30 17:36:39.846795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.846771 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1dfb448-5107-40ae-a8a7-6766274ad9ce-plugins-config-volume" pod "b1dfb448-5107-40ae-a8a7-6766274ad9ce" (UID: "b1dfb448-5107-40ae-a8a7-6766274ad9ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:39.847053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.847037 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1dfb448-5107-40ae-a8a7-6766274ad9ce-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:39.848538 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.848515 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1dfb448-5107-40ae-a8a7-6766274ad9ce-kube-api-access-9b9f8" pod "b1dfb448-5107-40ae-a8a7-6766274ad9ce" (UID: "b1dfb448-5107-40ae-a8a7-6766274ad9ce"). InnerVolumeSpecName "kube-api-access-9b9f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:39.948340 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:39.948305 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b9f8\" (UniqueName: \"kubernetes.io/projected/b1dfb448-5107-40ae-a8a7-6766274ad9ce-kube-api-access-9b9f8\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.122333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.122292 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:40.176638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.176606 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:36:40.196837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.196789 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" containerID="cri-o://aa4696d145d000c278f6b32801c26fc71baa181a26a9070bdd1cb138be78c2f4" gracePeriod=45 Oct 30 17:36:40.252623 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.252584 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f119ba5-b435-453d-809e-bb2795e3066d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f119ba5-b435-453d-809e-bb2795e3066d-istiod-ca-cert\") pod \"6f119ba5-b435-453d-809e-bb2795e3066d\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " Oct 30 17:36:40.253293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.252667 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-istio-token\" (UniqueName: \"kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-istio-token\") pod \"6f119ba5-b435-453d-809e-bb2795e3066d\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " Oct 30 17:36:40.253293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.252769 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-kube-api-access-4kdln\" (UniqueName: \"kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-kube-api-access-4kdln\") pod \"6f119ba5-b435-453d-809e-bb2795e3066d\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " Oct 30 17:36:40.253293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.252835 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-data\") pod \"6f119ba5-b435-453d-809e-bb2795e3066d\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " Oct 30 17:36:40.253293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.252927 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-certs\") pod \"6f119ba5-b435-453d-809e-bb2795e3066d\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " Oct 30 17:36:40.253763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.253423 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f119ba5-b435-453d-809e-bb2795e3066d-istiod-ca-cert" pod "6f119ba5-b435-453d-809e-bb2795e3066d" (UID: "6f119ba5-b435-453d-809e-bb2795e3066d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:40.253763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.253478 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-envoy\") pod \"6f119ba5-b435-453d-809e-bb2795e3066d\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " Oct 30 17:36:40.253763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.253572 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-credential-socket\") pod \"6f119ba5-b435-453d-809e-bb2795e3066d\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " Oct 30 17:36:40.253763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.253709 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f119ba5-b435-453d-809e-bb2795e3066d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f119ba5-b435-453d-809e-bb2795e3066d-istio-podinfo\") pod \"6f119ba5-b435-453d-809e-bb2795e3066d\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " Oct 30 17:36:40.254113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.253784 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-socket\") pod \"6f119ba5-b435-453d-809e-bb2795e3066d\" (UID: \"6f119ba5-b435-453d-809e-bb2795e3066d\") " Oct 30 17:36:40.254113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.253841 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-data" pod "6f119ba5-b435-453d-809e-bb2795e3066d" (UID: "6f119ba5-b435-453d-809e-bb2795e3066d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.254113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.253799 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-certs" pod "6f119ba5-b435-453d-809e-bb2795e3066d" (UID: "6f119ba5-b435-453d-809e-bb2795e3066d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.254341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.254118 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-socket" pod "6f119ba5-b435-453d-809e-bb2795e3066d" (UID: "6f119ba5-b435-453d-809e-bb2795e3066d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.254600 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.254579 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.254661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.254607 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.254661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.254618 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.254661 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.254630 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f119ba5-b435-453d-809e-bb2795e3066d-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.256179 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.256006 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-istio-token" pod "6f119ba5-b435-453d-809e-bb2795e3066d" (UID: "6f119ba5-b435-453d-809e-bb2795e3066d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:40.257086 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.257060 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-kube-api-access-4kdln" pod "6f119ba5-b435-453d-809e-bb2795e3066d" (UID: "6f119ba5-b435-453d-809e-bb2795e3066d"). InnerVolumeSpecName "kube-api-access-4kdln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:40.259053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.259025 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-credential-socket" pod "6f119ba5-b435-453d-809e-bb2795e3066d" (UID: "6f119ba5-b435-453d-809e-bb2795e3066d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.260760 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.260734 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-envoy" pod "6f119ba5-b435-453d-809e-bb2795e3066d" (UID: "6f119ba5-b435-453d-809e-bb2795e3066d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.261559 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.261537 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f119ba5-b435-453d-809e-bb2795e3066d-istio-podinfo" pod "6f119ba5-b435-453d-809e-bb2795e3066d" (UID: "6f119ba5-b435-453d-809e-bb2795e3066d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:36:40.295911 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.295814 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:40.418976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.404527 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.418976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.404558 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kdln\" (UniqueName: \"kubernetes.io/projected/6f119ba5-b435-453d-809e-bb2795e3066d-kube-api-access-4kdln\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.418976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.404574 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.418976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.404589 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f119ba5-b435-453d-809e-bb2795e3066d-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.418976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.404601 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f119ba5-b435-453d-809e-bb2795e3066d-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.539977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.539923 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:40.546932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.546549 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": dial tcp 10.132.0.79:8000: connect: connection refused" Oct 30 17:36:40.552620 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.552578 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p"] Oct 30 17:36:40.753178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.753139 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:36:40.822101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.822060 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tls-certs\") pod \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " Oct 30 17:36:40.822312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.822108 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tmp-dir\") pod \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " Oct 30 17:36:40.822312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.822149 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-model-cache\") pod \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " Oct 30 17:36:40.822312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.822204 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-dshm\") pod \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " Oct 30 17:36:40.822514 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.822372 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-kube-api-access-bg495\" (UniqueName: \"kubernetes.io/projected/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-kube-api-access-bg495\") pod \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " Oct 30 17:36:40.822514 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.822432 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-home\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-home\") pod \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\" (UID: \"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f\") " Oct 30 17:36:40.822514 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.822497 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tmp-dir" pod "abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" (UID: "abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.822689 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.822651 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-home" pod "abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" (UID: "abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.822844 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.822817 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-model-cache" pod "abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" (UID: "abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.823141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.823122 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.823226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.823146 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.823226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.823161 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.825511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.825337 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tls-certs" pod "abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" (UID: "abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:40.825969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.825892 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-kube-api-access-bg495" pod "abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" (UID: "abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f"). InnerVolumeSpecName "kube-api-access-bg495". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:40.826693 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.826666 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-dshm" pod "abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" (UID: "abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:40.847806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.847778 4828 generic.go:356] "Generic (PLEG): container finished" podID="6f119ba5-b435-453d-809e-bb2795e3066d" containerID="e8b359a17219267c2d4c5c0c79203547bd088ca2ccd224f9264d8226ac735b85" exitCode=0 Oct 30 17:36:40.847996 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.847986 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" event={"ID":"6f119ba5-b435-453d-809e-bb2795e3066d","Type":"ContainerDied","Data":"e8b359a17219267c2d4c5c0c79203547bd088ca2ccd224f9264d8226ac735b85"} Oct 30 17:36:40.848088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.848027 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" event={"ID":"6f119ba5-b435-453d-809e-bb2795e3066d","Type":"ContainerDied","Data":"e297c1ffb88504af7167eae8f6d7d5baa7f2d7ea6c16dd57d17a09d14d9f6aa0"} Oct 30 17:36:40.848088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.848048 4828 scope.go:122] "RemoveContainer" containerID="e8b359a17219267c2d4c5c0c79203547bd088ca2ccd224f9264d8226ac735b85" Oct 30 17:36:40.848212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.848177 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2" Oct 30 17:36:40.851073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.850641 4828 generic.go:356] "Generic (PLEG): container finished" podID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerID="aa4696d145d000c278f6b32801c26fc71baa181a26a9070bdd1cb138be78c2f4" exitCode=0 Oct 30 17:36:40.851073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.850756 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js" Oct 30 17:36:40.851422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.851390 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" Oct 30 17:36:40.854187 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.853870 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" event={"ID":"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f","Type":"ContainerDied","Data":"aa4696d145d000c278f6b32801c26fc71baa181a26a9070bdd1cb138be78c2f4"} Oct 30 17:36:40.854187 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.853948 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m" event={"ID":"abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f","Type":"ContainerDied","Data":"ebcad0ecb256990a0192db2a50899ebb9ca3fa7c5a6d36bcf58dcd8c3da0f6b8"} Oct 30 17:36:40.867642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.867620 4828 scope.go:122] "RemoveContainer" containerID="e4cc0b0e5c1b171b36712679a8469dbc926573876fdf65a825863e48f106f20f" Oct 30 17:36:40.881709 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.881684 4828 scope.go:122] "RemoveContainer" containerID="e8b359a17219267c2d4c5c0c79203547bd088ca2ccd224f9264d8226ac735b85" Oct 30 17:36:40.882130 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:40.882100 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b359a17219267c2d4c5c0c79203547bd088ca2ccd224f9264d8226ac735b85\": container with ID starting with e8b359a17219267c2d4c5c0c79203547bd088ca2ccd224f9264d8226ac735b85 not found: ID does not exist" containerID="e8b359a17219267c2d4c5c0c79203547bd088ca2ccd224f9264d8226ac735b85" Oct 30 17:36:40.882251 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.882143 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b359a17219267c2d4c5c0c79203547bd088ca2ccd224f9264d8226ac735b85"} err="failed to get container status \"e8b359a17219267c2d4c5c0c79203547bd088ca2ccd224f9264d8226ac735b85\": rpc error: code = NotFound desc = could not find container \"e8b359a17219267c2d4c5c0c79203547bd088ca2ccd224f9264d8226ac735b85\": container with ID starting with e8b359a17219267c2d4c5c0c79203547bd088ca2ccd224f9264d8226ac735b85 not found: ID does not exist" Oct 30 17:36:40.882251 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.882169 4828 scope.go:122] "RemoveContainer" containerID="e4cc0b0e5c1b171b36712679a8469dbc926573876fdf65a825863e48f106f20f" Oct 30 17:36:40.882506 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:40.882474 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4cc0b0e5c1b171b36712679a8469dbc926573876fdf65a825863e48f106f20f\": container with ID starting with e4cc0b0e5c1b171b36712679a8469dbc926573876fdf65a825863e48f106f20f not found: ID does not exist" containerID="e4cc0b0e5c1b171b36712679a8469dbc926573876fdf65a825863e48f106f20f" Oct 30 17:36:40.882619 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.882511 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4cc0b0e5c1b171b36712679a8469dbc926573876fdf65a825863e48f106f20f"} err="failed to get container status \"e4cc0b0e5c1b171b36712679a8469dbc926573876fdf65a825863e48f106f20f\": rpc error: code = NotFound desc = could not find container \"e4cc0b0e5c1b171b36712679a8469dbc926573876fdf65a825863e48f106f20f\": container with ID starting with e4cc0b0e5c1b171b36712679a8469dbc926573876fdf65a825863e48f106f20f not found: ID does not exist" Oct 30 17:36:40.882619 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.882532 4828 scope.go:122] "RemoveContainer" containerID="aa4696d145d000c278f6b32801c26fc71baa181a26a9070bdd1cb138be78c2f4" Oct 30 17:36:40.893716 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.893692 4828 scope.go:122] "RemoveContainer" containerID="aa4696d145d000c278f6b32801c26fc71baa181a26a9070bdd1cb138be78c2f4" Oct 30 17:36:40.894223 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:40.894158 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4696d145d000c278f6b32801c26fc71baa181a26a9070bdd1cb138be78c2f4\": container with ID starting with aa4696d145d000c278f6b32801c26fc71baa181a26a9070bdd1cb138be78c2f4 not found: ID does not exist" containerID="aa4696d145d000c278f6b32801c26fc71baa181a26a9070bdd1cb138be78c2f4" Oct 30 17:36:40.894223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.894187 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4696d145d000c278f6b32801c26fc71baa181a26a9070bdd1cb138be78c2f4"} err="failed to get container status \"aa4696d145d000c278f6b32801c26fc71baa181a26a9070bdd1cb138be78c2f4\": rpc error: code = NotFound desc = could not find container \"aa4696d145d000c278f6b32801c26fc71baa181a26a9070bdd1cb138be78c2f4\": container with ID starting with aa4696d145d000c278f6b32801c26fc71baa181a26a9070bdd1cb138be78c2f4 not found: ID does not exist" Oct 30 17:36:40.912402 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.912335 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2"] Oct 30 17:36:40.915992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.915965 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-db8417-openshift-default-68b9c867f9-524k2"] Oct 30 17:36:40.924710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.924687 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg495\" (UniqueName: \"kubernetes.io/projected/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-kube-api-access-bg495\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.924824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.924712 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.924824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.924724 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:40.949292 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.949264 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js"] Oct 30 17:36:40.953143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.953123 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-04248e-6c6cb74fd6-c65js"] Oct 30 17:36:40.989362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.989268 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m"] Oct 30 17:36:40.994704 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:40.994679 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b4f406/premium-w2-b4f406-kserve-77d46c6c94-qc99m"] Oct 30 17:36:41.432719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:41.429792 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:42.495205 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:42.495168 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" path="/var/lib/kubelet/pods/6f119ba5-b435-453d-809e-bb2795e3066d/volumes" Oct 30 17:36:42.495892 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:42.495871 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" path="/var/lib/kubelet/pods/abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f/volumes" Oct 30 17:36:42.496437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:42.496419 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1dfb448-5107-40ae-a8a7-6766274ad9ce" path="/var/lib/kubelet/pods/b1dfb448-5107-40ae-a8a7-6766274ad9ce/volumes" Oct 30 17:36:43.385641 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:43.385471 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:45.315731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:45.315686 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:45.811804 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:45.811765 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl"] Oct 30 17:36:45.846830 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:45.846781 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:46.009517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:46.009477 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:46.160056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:46.159968 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:47.484571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:47.484537 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv"] Oct 30 17:36:47.485114 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:47.484786 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" podUID="7a96b22c-8739-43ea-9b55-5ce228ad886a" containerName="maas-api" containerID="cri-o://4ab918c9074b8180cdb74a39e7e970059a39ca222541113af712e5e6d846ac49" gracePeriod=30 Oct 30 17:36:47.600195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:47.600159 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:47.844439 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:47.844406 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:36:47.982025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:47.981317 4828 generic.go:356] "Generic (PLEG): container finished" podID="7a96b22c-8739-43ea-9b55-5ce228ad886a" containerID="4ab918c9074b8180cdb74a39e7e970059a39ca222541113af712e5e6d846ac49" exitCode=0 Oct 30 17:36:47.982025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:47.981416 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" event={"ID":"7a96b22c-8739-43ea-9b55-5ce228ad886a","Type":"ContainerDied","Data":"4ab918c9074b8180cdb74a39e7e970059a39ca222541113af712e5e6d846ac49"} Oct 30 17:36:47.982025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:47.981466 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" event={"ID":"7a96b22c-8739-43ea-9b55-5ce228ad886a","Type":"ContainerDied","Data":"c52e2b6b69f393ef735b71b2cebb69d72e550843ea62e0b678ea9704da9cb806"} Oct 30 17:36:47.982025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:47.981500 4828 scope.go:122] "RemoveContainer" containerID="4ab918c9074b8180cdb74a39e7e970059a39ca222541113af712e5e6d846ac49" Oct 30 17:36:47.982025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:47.981646 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv" Oct 30 17:36:47.996220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:47.996179 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-maas-api-tls\") pod \"7a96b22c-8739-43ea-9b55-5ce228ad886a\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " Oct 30 17:36:47.996691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:47.996661 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-metrics-certs\") pod \"7a96b22c-8739-43ea-9b55-5ce228ad886a\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " Oct 30 17:36:47.996889 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:47.996864 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a96b22c-8739-43ea-9b55-5ce228ad886a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a96b22c-8739-43ea-9b55-5ce228ad886a-openshift-service-ca\") pod \"7a96b22c-8739-43ea-9b55-5ce228ad886a\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " Oct 30 17:36:47.997065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:47.997045 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a96b22c-8739-43ea-9b55-5ce228ad886a-kube-api-access-kv2md\" (UniqueName: \"kubernetes.io/projected/7a96b22c-8739-43ea-9b55-5ce228ad886a-kube-api-access-kv2md\") pod \"7a96b22c-8739-43ea-9b55-5ce228ad886a\" (UID: \"7a96b22c-8739-43ea-9b55-5ce228ad886a\") " Oct 30 17:36:48.000301 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.000216 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a96b22c-8739-43ea-9b55-5ce228ad886a-openshift-service-ca" pod "7a96b22c-8739-43ea-9b55-5ce228ad886a" (UID: "7a96b22c-8739-43ea-9b55-5ce228ad886a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:48.016286 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.016001 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-metrics-certs" pod "7a96b22c-8739-43ea-9b55-5ce228ad886a" (UID: "7a96b22c-8739-43ea-9b55-5ce228ad886a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:48.037955 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.037538 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a96b22c-8739-43ea-9b55-5ce228ad886a-kube-api-access-kv2md" pod "7a96b22c-8739-43ea-9b55-5ce228ad886a" (UID: "7a96b22c-8739-43ea-9b55-5ce228ad886a"). InnerVolumeSpecName "kube-api-access-kv2md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:48.039701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.039653 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-maas-api-tls" pod "7a96b22c-8739-43ea-9b55-5ce228ad886a" (UID: "7a96b22c-8739-43ea-9b55-5ce228ad886a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:48.046872 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.046837 4828 scope.go:122] "RemoveContainer" containerID="4ab918c9074b8180cdb74a39e7e970059a39ca222541113af712e5e6d846ac49" Oct 30 17:36:48.047269 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:48.047247 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab918c9074b8180cdb74a39e7e970059a39ca222541113af712e5e6d846ac49\": container with ID starting with 4ab918c9074b8180cdb74a39e7e970059a39ca222541113af712e5e6d846ac49 not found: ID does not exist" containerID="4ab918c9074b8180cdb74a39e7e970059a39ca222541113af712e5e6d846ac49" Oct 30 17:36:48.047350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.047282 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab918c9074b8180cdb74a39e7e970059a39ca222541113af712e5e6d846ac49"} err="failed to get container status \"4ab918c9074b8180cdb74a39e7e970059a39ca222541113af712e5e6d846ac49\": rpc error: code = NotFound desc = could not find container \"4ab918c9074b8180cdb74a39e7e970059a39ca222541113af712e5e6d846ac49\": container with ID starting with 4ab918c9074b8180cdb74a39e7e970059a39ca222541113af712e5e6d846ac49 not found: ID does not exist" Oct 30 17:36:48.099233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.099134 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:48.099233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.099170 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a96b22c-8739-43ea-9b55-5ce228ad886a-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:48.099233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.099181 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv2md\" (UniqueName: \"kubernetes.io/projected/7a96b22c-8739-43ea-9b55-5ce228ad886a-kube-api-access-kv2md\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:48.099233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.099191 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a96b22c-8739-43ea-9b55-5ce228ad886a-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:48.163813 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.163781 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:48.335466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.335430 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv"] Oct 30 17:36:48.335813 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:48.335790 4828 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a96b22c_8739_43ea_9b55_5ce228ad886a.slice\": RecentStats: unable to find data in memory cache]" Oct 30 17:36:48.346793 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.346754 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:48.348491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.348465 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-82243b-6c5c8c6576-lpgqv"] Oct 30 17:36:48.509003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.494714 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:48.528151 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.528110 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a96b22c-8739-43ea-9b55-5ce228ad886a" path="/var/lib/kubelet/pods/7a96b22c-8739-43ea-9b55-5ce228ad886a/volumes" Oct 30 17:36:48.661976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.661893 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:48.817017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:48.816895 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:49.670698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:49.670660 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:49.835708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:49.835670 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:50.772124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:50.769059 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:50.812597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:50.812545 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" podUID="c4eb9078-46ec-4fa4-b0a6-dc78e060859e" containerName="payload-pre-processing" containerID="cri-o://daeca9c816e887a99e2adf62b95eefb00178099c5e06b1c4ee607eeae761c4b1" gracePeriod=25 Oct 30 17:36:52.734088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:52.732286 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:54.351177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:54.348490 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:55.595843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:55.595790 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" containerID="cri-o://a0eaa42278795a387e4c1a26b484906aa6b0c214823d248c8a2191d1668b4fb8" gracePeriod=45 Oct 30 17:36:55.664874 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:55.664835 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:55.837564 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:55.837523 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:55.944246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:55.944216 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:36:56.019334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.019291 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-home\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-home\") pod \"bddf0988-e42b-43c1-80a8-e22d04bdb133\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " Oct 30 17:36:56.019541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.019357 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-dshm\") pod \"bddf0988-e42b-43c1-80a8-e22d04bdb133\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " Oct 30 17:36:56.019541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.019404 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bddf0988-e42b-43c1-80a8-e22d04bdb133-tls-certs\" (UniqueName: \"kubernetes.io/secret/bddf0988-e42b-43c1-80a8-e22d04bdb133-tls-certs\") pod \"bddf0988-e42b-43c1-80a8-e22d04bdb133\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " Oct 30 17:36:56.020685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.019859 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bddf0988-e42b-43c1-80a8-e22d04bdb133-kube-api-access-j8sr4\" (UniqueName: \"kubernetes.io/projected/bddf0988-e42b-43c1-80a8-e22d04bdb133-kube-api-access-j8sr4\") pod \"bddf0988-e42b-43c1-80a8-e22d04bdb133\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " Oct 30 17:36:56.020685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.020021 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-tmp-dir\") pod \"bddf0988-e42b-43c1-80a8-e22d04bdb133\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " Oct 30 17:36:56.020685 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.020088 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-model-cache\") pod \"bddf0988-e42b-43c1-80a8-e22d04bdb133\" (UID: \"bddf0988-e42b-43c1-80a8-e22d04bdb133\") " Oct 30 17:36:56.020964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.020885 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-model-cache" pod "bddf0988-e42b-43c1-80a8-e22d04bdb133" (UID: "bddf0988-e42b-43c1-80a8-e22d04bdb133"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:56.021029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.021000 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-tmp-dir" pod "bddf0988-e42b-43c1-80a8-e22d04bdb133" (UID: "bddf0988-e42b-43c1-80a8-e22d04bdb133"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:56.021092 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.021021 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-home" pod "bddf0988-e42b-43c1-80a8-e22d04bdb133" (UID: "bddf0988-e42b-43c1-80a8-e22d04bdb133"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:56.022757 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.022727 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-dshm" pod "bddf0988-e42b-43c1-80a8-e22d04bdb133" (UID: "bddf0988-e42b-43c1-80a8-e22d04bdb133"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:36:56.023288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.023258 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bddf0988-e42b-43c1-80a8-e22d04bdb133-kube-api-access-j8sr4" pod "bddf0988-e42b-43c1-80a8-e22d04bdb133" (UID: "bddf0988-e42b-43c1-80a8-e22d04bdb133"). InnerVolumeSpecName "kube-api-access-j8sr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:56.023371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.023277 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bddf0988-e42b-43c1-80a8-e22d04bdb133-tls-certs" pod "bddf0988-e42b-43c1-80a8-e22d04bdb133" (UID: "bddf0988-e42b-43c1-80a8-e22d04bdb133"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:56.094468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.094426 4828 generic.go:356] "Generic (PLEG): container finished" podID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerID="a0eaa42278795a387e4c1a26b484906aa6b0c214823d248c8a2191d1668b4fb8" exitCode=0 Oct 30 17:36:56.094682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.094484 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" Oct 30 17:36:56.094682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.094509 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" event={"ID":"bddf0988-e42b-43c1-80a8-e22d04bdb133","Type":"ContainerDied","Data":"a0eaa42278795a387e4c1a26b484906aa6b0c214823d248c8a2191d1668b4fb8"} Oct 30 17:36:56.094682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.094578 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p" event={"ID":"bddf0988-e42b-43c1-80a8-e22d04bdb133","Type":"ContainerDied","Data":"2abeb0faddcb3fe1a60c37d7ce5e9c5d8d430e15958aba028b1b0893b3564646"} Oct 30 17:36:56.094682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.094603 4828 scope.go:122] "RemoveContainer" containerID="a0eaa42278795a387e4c1a26b484906aa6b0c214823d248c8a2191d1668b4fb8" Oct 30 17:36:56.109665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.109639 4828 scope.go:122] "RemoveContainer" containerID="a0eaa42278795a387e4c1a26b484906aa6b0c214823d248c8a2191d1668b4fb8" Oct 30 17:36:56.110023 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:56.110001 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0eaa42278795a387e4c1a26b484906aa6b0c214823d248c8a2191d1668b4fb8\": container with ID starting with a0eaa42278795a387e4c1a26b484906aa6b0c214823d248c8a2191d1668b4fb8 not found: ID does not exist" containerID="a0eaa42278795a387e4c1a26b484906aa6b0c214823d248c8a2191d1668b4fb8" Oct 30 17:36:56.110119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.110037 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0eaa42278795a387e4c1a26b484906aa6b0c214823d248c8a2191d1668b4fb8"} err="failed to get container status \"a0eaa42278795a387e4c1a26b484906aa6b0c214823d248c8a2191d1668b4fb8\": rpc error: code = NotFound desc = could not find container \"a0eaa42278795a387e4c1a26b484906aa6b0c214823d248c8a2191d1668b4fb8\": container with ID starting with a0eaa42278795a387e4c1a26b484906aa6b0c214823d248c8a2191d1668b4fb8 not found: ID does not exist" Oct 30 17:36:56.122484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.122452 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.122484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.122482 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.122484 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.122491 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.122766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.122505 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bddf0988-e42b-43c1-80a8-e22d04bdb133-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.122766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.122519 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bddf0988-e42b-43c1-80a8-e22d04bdb133-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.122766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.122531 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8sr4\" (UniqueName: \"kubernetes.io/projected/bddf0988-e42b-43c1-80a8-e22d04bdb133-kube-api-access-j8sr4\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:56.150790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.150756 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p"] Oct 30 17:36:56.156506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.156427 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b80d83/distinct-2-w3-b80d83-kserve-5bbbfb9f5d-cmz5p"] Oct 30 17:36:56.229368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.229325 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:56.380549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.380485 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:56.494387 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.494299 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" path="/var/lib/kubelet/pods/bddf0988-e42b-43c1-80a8-e22d04bdb133/volumes" Oct 30 17:36:56.589523 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:56.589487 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:57.127274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:57.127235 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:57.683174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:57.683133 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf"] Oct 30 17:36:57.683468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:57.683438 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" podUID="14d34d00-c34b-4a89-bc00-06d8972e0ab0" containerName="maas-api" containerID="cri-o://f4f04bdf93064626e7d68814f0e8f412ab9be8c10aef681440f6e19f42fee773" gracePeriod=30 Oct 30 17:36:57.782753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:57.782719 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:58.031727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.031703 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 17:36:58.171597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.170298 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-maas-api-tls\") pod \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " Oct 30 17:36:58.171597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.170420 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d34d00-c34b-4a89-bc00-06d8972e0ab0-kube-api-access-m2md9\" (UniqueName: \"kubernetes.io/projected/14d34d00-c34b-4a89-bc00-06d8972e0ab0-kube-api-access-m2md9\") pod \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " Oct 30 17:36:58.171597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.170480 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d34d00-c34b-4a89-bc00-06d8972e0ab0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d34d00-c34b-4a89-bc00-06d8972e0ab0-openshift-service-ca\") pod \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " Oct 30 17:36:58.171597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.170536 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-metrics-certs\") pod \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\" (UID: \"14d34d00-c34b-4a89-bc00-06d8972e0ab0\") " Oct 30 17:36:58.173919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.173878 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d34d00-c34b-4a89-bc00-06d8972e0ab0-openshift-service-ca" pod "14d34d00-c34b-4a89-bc00-06d8972e0ab0" (UID: "14d34d00-c34b-4a89-bc00-06d8972e0ab0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:36:58.176166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.176138 4828 generic.go:356] "Generic (PLEG): container finished" podID="14d34d00-c34b-4a89-bc00-06d8972e0ab0" containerID="f4f04bdf93064626e7d68814f0e8f412ab9be8c10aef681440f6e19f42fee773" exitCode=0 Oct 30 17:36:58.176588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.176546 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" event={"ID":"14d34d00-c34b-4a89-bc00-06d8972e0ab0","Type":"ContainerDied","Data":"f4f04bdf93064626e7d68814f0e8f412ab9be8c10aef681440f6e19f42fee773"} Oct 30 17:36:58.176781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.176760 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" event={"ID":"14d34d00-c34b-4a89-bc00-06d8972e0ab0","Type":"ContainerDied","Data":"fab725fa1de936e4184f6f841035ffe43522f6eea6baf1d31c573abfed9e2a1f"} Oct 30 17:36:58.176941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.176926 4828 scope.go:122] "RemoveContainer" containerID="f4f04bdf93064626e7d68814f0e8f412ab9be8c10aef681440f6e19f42fee773" Oct 30 17:36:58.177181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.177166 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf" Oct 30 17:36:58.215393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.205880 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d34d00-c34b-4a89-bc00-06d8972e0ab0-kube-api-access-m2md9" pod "14d34d00-c34b-4a89-bc00-06d8972e0ab0" (UID: "14d34d00-c34b-4a89-bc00-06d8972e0ab0"). InnerVolumeSpecName "kube-api-access-m2md9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:36:58.215393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.206141 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-metrics-certs" pod "14d34d00-c34b-4a89-bc00-06d8972e0ab0" (UID: "14d34d00-c34b-4a89-bc00-06d8972e0ab0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:58.215791 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.215758 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-maas-api-tls" pod "14d34d00-c34b-4a89-bc00-06d8972e0ab0" (UID: "14d34d00-c34b-4a89-bc00-06d8972e0ab0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:36:58.245425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.245282 4828 scope.go:122] "RemoveContainer" containerID="f4f04bdf93064626e7d68814f0e8f412ab9be8c10aef681440f6e19f42fee773" Oct 30 17:36:58.245877 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:36:58.245811 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f04bdf93064626e7d68814f0e8f412ab9be8c10aef681440f6e19f42fee773\": container with ID starting with f4f04bdf93064626e7d68814f0e8f412ab9be8c10aef681440f6e19f42fee773 not found: ID does not exist" containerID="f4f04bdf93064626e7d68814f0e8f412ab9be8c10aef681440f6e19f42fee773" Oct 30 17:36:58.245877 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.245847 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f04bdf93064626e7d68814f0e8f412ab9be8c10aef681440f6e19f42fee773"} err="failed to get container status \"f4f04bdf93064626e7d68814f0e8f412ab9be8c10aef681440f6e19f42fee773\": rpc error: code = NotFound desc = could not find container \"f4f04bdf93064626e7d68814f0e8f412ab9be8c10aef681440f6e19f42fee773\": container with ID starting with f4f04bdf93064626e7d68814f0e8f412ab9be8c10aef681440f6e19f42fee773 not found: ID does not exist" Oct 30 17:36:58.274007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.273961 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2md9\" (UniqueName: \"kubernetes.io/projected/14d34d00-c34b-4a89-bc00-06d8972e0ab0-kube-api-access-m2md9\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:58.274007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.274003 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d34d00-c34b-4a89-bc00-06d8972e0ab0-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:58.274227 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.274018 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:58.274227 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.274032 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d34d00-c34b-4a89-bc00-06d8972e0ab0-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:36:58.502617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.502530 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:58.534393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.534345 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf"] Oct 30 17:36:58.547629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.547595 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-04248e-7b694557d4-h5rvf"] Oct 30 17:36:58.658196 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.658154 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:58.808536 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:58.808489 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:36:59.193712 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:36:59.193615 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:00.205025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.204985 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" podUID="c4eb9078-46ec-4fa4-b0a6-dc78e060859e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:37:00.270113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.270074 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:00.493955 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.493827 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d34d00-c34b-4a89-bc00-06d8972e0ab0" path="/var/lib/kubelet/pods/14d34d00-c34b-4a89-bc00-06d8972e0ab0/volumes" Oct 30 17:37:00.961632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.961596 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg"] Oct 30 17:37:00.962288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962270 4828 cpu_manager.go:414] "containerMap: removing container" podUID="14d34d00-c34b-4a89-bc00-06d8972e0ab0" containerName="maas-api" Oct 30 17:37:00.962288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962289 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d34d00-c34b-4a89-bc00-06d8972e0ab0" containerName="maas-api" Oct 30 17:37:00.962411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962298 4828 cpu_manager.go:414] "containerMap: removing container" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" Oct 30 17:37:00.962411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962303 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" Oct 30 17:37:00.962411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962316 4828 cpu_manager.go:414] "containerMap: removing container" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" Oct 30 17:37:00.962411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962322 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" Oct 30 17:37:00.962411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962334 4828 cpu_manager.go:414] "containerMap: removing container" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" Oct 30 17:37:00.962411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962339 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" Oct 30 17:37:00.962411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962348 4828 cpu_manager.go:414] "containerMap: removing container" podUID="7a96b22c-8739-43ea-9b55-5ce228ad886a" containerName="maas-api" Oct 30 17:37:00.962411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962353 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a96b22c-8739-43ea-9b55-5ce228ad886a" containerName="maas-api" Oct 30 17:37:00.962411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962362 4828 cpu_manager.go:414] "containerMap: removing container" podUID="b1dfb448-5107-40ae-a8a7-6766274ad9ce" containerName="payload-processing" Oct 30 17:37:00.962411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962367 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1dfb448-5107-40ae-a8a7-6766274ad9ce" containerName="payload-processing" Oct 30 17:37:00.962749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962481 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" Oct 30 17:37:00.962749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962490 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d34d00-c34b-4a89-bc00-06d8972e0ab0" containerName="maas-api" Oct 30 17:37:00.962749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962497 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" Oct 30 17:37:00.962749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962504 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc0ea64-db5b-4cf9-abcf-cb18b4fadf1f" containerName="main" Oct 30 17:37:00.962749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962512 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="bddf0988-e42b-43c1-80a8-e22d04bdb133" containerName="main" Oct 30 17:37:00.962749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962520 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a96b22c-8739-43ea-9b55-5ce228ad886a" containerName="maas-api" Oct 30 17:37:00.962749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.962526 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1dfb448-5107-40ae-a8a7-6766274ad9ce" containerName="payload-processing" Oct 30 17:37:00.967676 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.967651 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:00.996811 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:00.996689 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-054acd-serving-cert\"" Oct 30 17:37:01.006241 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.006208 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mgt2\" (UniqueName: \"kubernetes.io/projected/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-kube-api-access-4mgt2\") pod \"maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:01.006437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.006264 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:01.006437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.006317 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:01.006437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.006328 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg"] Oct 30 17:37:01.006617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.006568 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-metrics-certs\") pod \"maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:01.083481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.083444 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:01.107289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.107246 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-metrics-certs\") pod \"maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:01.107509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.107310 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgt2\" (UniqueName: \"kubernetes.io/projected/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-kube-api-access-4mgt2\") pod \"maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:01.107509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.107358 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:01.107509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.107404 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:01.108305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.108272 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:01.109938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.109911 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:01.110053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.109941 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-metrics-certs\") pod \"maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:01.132248 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.132219 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mgt2\" (UniqueName: \"kubernetes.io/projected/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-kube-api-access-4mgt2\") pod \"maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:01.232392 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.232300 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:01.308219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.308171 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:01.622513 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:01.622479 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg"] Oct 30 17:37:01.622872 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:37:01.622847 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d6e315_e9a3_47d1_a4b3_61b43c3d611b.slice/crio-598ab3d4c44fbcb2da711969b4ce8000c5b7e59c6368622c43ffc610e8fa5a25 WatchSource:0}: Error finding container 598ab3d4c44fbcb2da711969b4ce8000c5b7e59c6368622c43ffc610e8fa5a25: Status 404 returned error can't find the container with id 598ab3d4c44fbcb2da711969b4ce8000c5b7e59c6368622c43ffc610e8fa5a25 Oct 30 17:37:02.031397 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:02.031367 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z"] Oct 30 17:37:02.206730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:02.206634 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:02.248727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:02.247421 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" event={"ID":"95d6e315-e9a3-47d1-a4b3-61b43c3d611b","Type":"ContainerStarted","Data":"3376a24c250f96f22d3d5416b5bbcfecb63e6cce1032c3a3202e00d6ba318892"} Oct 30 17:37:02.248727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:02.247490 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" event={"ID":"95d6e315-e9a3-47d1-a4b3-61b43c3d611b","Type":"ContainerStarted","Data":"598ab3d4c44fbcb2da711969b4ce8000c5b7e59c6368622c43ffc610e8fa5a25"} Oct 30 17:37:02.248727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:02.248577 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:02.292920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:02.292701 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" podStartSLOduration=1.955131756 podStartE2EDuration="2.292681302s" totalImagesPullingTime="337.549546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:37:02.291563252 +0000 UTC m=+2999.937073432" watchObservedRunningTime="2026-10-30 17:37:02.292681302 +0000 UTC m=+2999.938191483" Oct 30 17:37:02.854411 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:02.854350 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:03.549038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:03.549002 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:03.730272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:03.730227 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:04.515200 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:04.513951 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:04.803639 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:04.803531 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:06.971931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:06.971270 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:07.215663 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:07.215626 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:07.372033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:07.371996 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq"] Oct 30 17:37:07.373137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:07.373104 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" containerID="cri-o://a04e662daadb2596a48ca94b7233e465579e84f5adf83e326bbce3612e95305b" gracePeriod=30 Oct 30 17:37:07.434816 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:07.434780 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf"] Oct 30 17:37:07.549921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:07.537956 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh"] Oct 30 17:37:07.921895 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:37:07.921856 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:37:07.929394 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:07.929344 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:37:07.934072 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:07.934047 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:37:08.215977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:08.210043 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:09.132963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:09.132917 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:09.269337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:09.269304 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:37:09.530079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:09.530041 4828 patch_prober.go:29] interesting pod/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:37:09.530286 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:09.530095 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:37:10.204456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:10.204416 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" podUID="c4eb9078-46ec-4fa4-b0a6-dc78e060859e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:37:10.623100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:10.623064 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:10.794002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:10.793959 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:11.089247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.089219 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" Oct 30 17:37:11.171881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.171839 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-kube-api-access-z8b6c\" (UniqueName: \"kubernetes.io/projected/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-kube-api-access-z8b6c\") pod \"c4eb9078-46ec-4fa4-b0a6-dc78e060859e\" (UID: \"c4eb9078-46ec-4fa4-b0a6-dc78e060859e\") " Oct 30 17:37:11.172274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.172000 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-plugins-config-volume\") pod \"c4eb9078-46ec-4fa4-b0a6-dc78e060859e\" (UID: \"c4eb9078-46ec-4fa4-b0a6-dc78e060859e\") " Oct 30 17:37:11.172426 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.172397 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-plugins-config-volume" pod "c4eb9078-46ec-4fa4-b0a6-dc78e060859e" (UID: "c4eb9078-46ec-4fa4-b0a6-dc78e060859e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:11.173306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.173277 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:11.174854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.174793 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-kube-api-access-z8b6c" pod "c4eb9078-46ec-4fa4-b0a6-dc78e060859e" (UID: "c4eb9078-46ec-4fa4-b0a6-dc78e060859e"). InnerVolumeSpecName "kube-api-access-z8b6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:11.275475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.275427 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8b6c\" (UniqueName: \"kubernetes.io/projected/c4eb9078-46ec-4fa4-b0a6-dc78e060859e-kube-api-access-z8b6c\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:11.323597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.323558 4828 generic.go:356] "Generic (PLEG): container finished" podID="c4eb9078-46ec-4fa4-b0a6-dc78e060859e" containerID="daeca9c816e887a99e2adf62b95eefb00178099c5e06b1c4ee607eeae761c4b1" exitCode=0 Oct 30 17:37:11.323771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.323611 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" Oct 30 17:37:11.323771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.323649 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" event={"ID":"c4eb9078-46ec-4fa4-b0a6-dc78e060859e","Type":"ContainerDied","Data":"daeca9c816e887a99e2adf62b95eefb00178099c5e06b1c4ee607eeae761c4b1"} Oct 30 17:37:11.323771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.323710 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl" event={"ID":"c4eb9078-46ec-4fa4-b0a6-dc78e060859e","Type":"ContainerDied","Data":"2bb6cfb2c75858b72bc4147cb82d3f48a7c08a13e729086921e834a07b3b2eb6"} Oct 30 17:37:11.323771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.323735 4828 scope.go:122] "RemoveContainer" containerID="daeca9c816e887a99e2adf62b95eefb00178099c5e06b1c4ee607eeae761c4b1" Oct 30 17:37:11.335014 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.334994 4828 scope.go:122] "RemoveContainer" containerID="daeca9c816e887a99e2adf62b95eefb00178099c5e06b1c4ee607eeae761c4b1" Oct 30 17:37:11.335337 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:37:11.335312 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daeca9c816e887a99e2adf62b95eefb00178099c5e06b1c4ee607eeae761c4b1\": container with ID starting with daeca9c816e887a99e2adf62b95eefb00178099c5e06b1c4ee607eeae761c4b1 not found: ID does not exist" containerID="daeca9c816e887a99e2adf62b95eefb00178099c5e06b1c4ee607eeae761c4b1" Oct 30 17:37:11.335415 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.335349 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daeca9c816e887a99e2adf62b95eefb00178099c5e06b1c4ee607eeae761c4b1"} err="failed to get container status \"daeca9c816e887a99e2adf62b95eefb00178099c5e06b1c4ee607eeae761c4b1\": rpc error: code = NotFound desc = could not find container \"daeca9c816e887a99e2adf62b95eefb00178099c5e06b1c4ee607eeae761c4b1\": container with ID starting with daeca9c816e887a99e2adf62b95eefb00178099c5e06b1c4ee607eeae761c4b1 not found: ID does not exist" Oct 30 17:37:11.383944 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.383888 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl"] Oct 30 17:37:11.387643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:11.387615 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b80d83-6b78bf564c-vd2sl"] Oct 30 17:37:12.436202 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.436129 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" podUID="666617d1-e514-4b17-ae65-1da9dfeea316" containerName="payload-processing" containerID="cri-o://234edcfaa0fd28d71f3e68f93d2456a12083e270067bf98e5d652e0ca18c9588" gracePeriod=25 Oct 30 17:37:12.492103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.492079 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4eb9078-46ec-4fa4-b0a6-dc78e060859e" path="/var/lib/kubelet/pods/c4eb9078-46ec-4fa4-b0a6-dc78e060859e/volumes" Oct 30 17:37:12.538665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.538624 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" podUID="14d8d98a-d3b2-4c79-9ebe-d9f817498f61" containerName="payload-pre-processing" containerID="cri-o://336b6335ef8493e9436208e409aca31f68e807b9e164599ede7bc7e7e1bf06aa" gracePeriod=25 Oct 30 17:37:12.694409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.694379 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 17:37:12.797038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.796991 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-podinfo\") pod \"05de1e06-0e9b-46a5-9020-5d791da7654a\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " Oct 30 17:37:12.797247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.797138 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-certs\") pod \"05de1e06-0e9b-46a5-9020-5d791da7654a\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " Oct 30 17:37:12.797247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.797186 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05de1e06-0e9b-46a5-9020-5d791da7654a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05de1e06-0e9b-46a5-9020-5d791da7654a-istiod-ca-cert\") pod \"05de1e06-0e9b-46a5-9020-5d791da7654a\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " Oct 30 17:37:12.797381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.797255 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-kube-api-access-s67sd\" (UniqueName: \"kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-kube-api-access-s67sd\") pod \"05de1e06-0e9b-46a5-9020-5d791da7654a\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " Oct 30 17:37:12.797381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.797295 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-credential-socket\") pod \"05de1e06-0e9b-46a5-9020-5d791da7654a\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " Oct 30 17:37:12.797381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.797372 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-data\") pod \"05de1e06-0e9b-46a5-9020-5d791da7654a\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " Oct 30 17:37:12.797561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.797457 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-envoy\") pod \"05de1e06-0e9b-46a5-9020-5d791da7654a\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " Oct 30 17:37:12.797561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.797484 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-token\" (UniqueName: \"kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-token\") pod \"05de1e06-0e9b-46a5-9020-5d791da7654a\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " Oct 30 17:37:12.797561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.797546 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-socket\") pod \"05de1e06-0e9b-46a5-9020-5d791da7654a\" (UID: \"05de1e06-0e9b-46a5-9020-5d791da7654a\") " Oct 30 17:37:12.798264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.798040 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-credential-socket" pod "05de1e06-0e9b-46a5-9020-5d791da7654a" (UID: "05de1e06-0e9b-46a5-9020-5d791da7654a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:12.798424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.798357 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-socket" pod "05de1e06-0e9b-46a5-9020-5d791da7654a" (UID: "05de1e06-0e9b-46a5-9020-5d791da7654a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:12.798424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.798385 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-certs" pod "05de1e06-0e9b-46a5-9020-5d791da7654a" (UID: "05de1e06-0e9b-46a5-9020-5d791da7654a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:12.798689 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.798601 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-data" pod "05de1e06-0e9b-46a5-9020-5d791da7654a" (UID: "05de1e06-0e9b-46a5-9020-5d791da7654a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:12.798689 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.798682 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05de1e06-0e9b-46a5-9020-5d791da7654a-istiod-ca-cert" pod "05de1e06-0e9b-46a5-9020-5d791da7654a" (UID: "05de1e06-0e9b-46a5-9020-5d791da7654a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:12.813893 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.813853 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-kube-api-access-s67sd" pod "05de1e06-0e9b-46a5-9020-5d791da7654a" (UID: "05de1e06-0e9b-46a5-9020-5d791da7654a"). InnerVolumeSpecName "kube-api-access-s67sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:12.814398 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.814317 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-envoy" pod "05de1e06-0e9b-46a5-9020-5d791da7654a" (UID: "05de1e06-0e9b-46a5-9020-5d791da7654a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:12.814506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.814409 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-token" pod "05de1e06-0e9b-46a5-9020-5d791da7654a" (UID: "05de1e06-0e9b-46a5-9020-5d791da7654a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:12.816823 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.816775 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-podinfo" pod "05de1e06-0e9b-46a5-9020-5d791da7654a" (UID: "05de1e06-0e9b-46a5-9020-5d791da7654a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:37:12.833466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.833436 4828 scope.go:122] "RemoveContainer" containerID="7e7972e3031c2ce34e5ec3e5b93f502d89aeb3035d4a4844d0123ba157569972" Oct 30 17:37:12.844664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.844640 4828 scope.go:122] "RemoveContainer" containerID="a04e662daadb2596a48ca94b7233e465579e84f5adf83e326bbce3612e95305b" Oct 30 17:37:12.854795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.854774 4828 scope.go:122] "RemoveContainer" containerID="a7013b65c85d127e323cc94864ba063869fc56618176f3432a6353aea62e59f3" Oct 30 17:37:12.899798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.899770 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:12.899798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.899799 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05de1e06-0e9b-46a5-9020-5d791da7654a-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:12.899968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.899809 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s67sd\" (UniqueName: \"kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-kube-api-access-s67sd\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:12.899968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.899820 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:12.899968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.899830 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:12.899968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.899843 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:12.899968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.899853 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:12.899968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.899861 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05de1e06-0e9b-46a5-9020-5d791da7654a-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:12.899968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:12.899869 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05de1e06-0e9b-46a5-9020-5d791da7654a-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:13.099703 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:13.098233 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:13.347140 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:13.347088 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" event={"ID":"05de1e06-0e9b-46a5-9020-5d791da7654a","Type":"ContainerDied","Data":"a04e662daadb2596a48ca94b7233e465579e84f5adf83e326bbce3612e95305b"} Oct 30 17:37:13.347338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:13.347168 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" event={"ID":"05de1e06-0e9b-46a5-9020-5d791da7654a","Type":"ContainerDied","Data":"7b29e02c4e68f008e96b08116e9b44722cf07aa2b2823eaf840d4dd855438d44"} Oct 30 17:37:13.347338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:13.347220 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq" Oct 30 17:37:13.436704 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:13.436664 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq"] Oct 30 17:37:13.443508 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:13.443477 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-04248e-openshift-default-7f694b565-9fcmq"] Oct 30 17:37:14.492277 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:14.492188 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" path="/var/lib/kubelet/pods/05de1e06-0e9b-46a5-9020-5d791da7654a/volumes" Oct 30 17:37:15.413052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:15.413014 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:17.059632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.059552 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" containerID="cri-o://43568ab0be19ea1ac8aaacc7a72253e196283fc6c64f4248d7a18c7964f7d2e2" gracePeriod=45 Oct 30 17:37:17.194247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.194020 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:17.427978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.427940 4828 generic.go:356] "Generic (PLEG): container finished" podID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerID="43568ab0be19ea1ac8aaacc7a72253e196283fc6c64f4248d7a18c7964f7d2e2" exitCode=0 Oct 30 17:37:17.428183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.428103 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" event={"ID":"2fb327ed-88b8-4f18-81b7-881fc4480618","Type":"ContainerDied","Data":"43568ab0be19ea1ac8aaacc7a72253e196283fc6c64f4248d7a18c7964f7d2e2"} Oct 30 17:37:17.428183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.428147 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" event={"ID":"2fb327ed-88b8-4f18-81b7-881fc4480618","Type":"ContainerDied","Data":"d54089e849ad7b7bc8e37268f22d902881e2dac00abb9ed0981d7d58f2d1e31c"} Oct 30 17:37:17.428183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.428163 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d54089e849ad7b7bc8e37268f22d902881e2dac00abb9ed0981d7d58f2d1e31c" Oct 30 17:37:17.459924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.456166 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:37:17.591921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.591544 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb327ed-88b8-4f18-81b7-881fc4480618-tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb327ed-88b8-4f18-81b7-881fc4480618-tls-certs\") pod \"2fb327ed-88b8-4f18-81b7-881fc4480618\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " Oct 30 17:37:17.591921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.591634 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-dshm\") pod \"2fb327ed-88b8-4f18-81b7-881fc4480618\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " Oct 30 17:37:17.591921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.591699 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb327ed-88b8-4f18-81b7-881fc4480618-kube-api-access-4h64p\" (UniqueName: \"kubernetes.io/projected/2fb327ed-88b8-4f18-81b7-881fc4480618-kube-api-access-4h64p\") pod \"2fb327ed-88b8-4f18-81b7-881fc4480618\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " Oct 30 17:37:17.591921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.591751 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-home\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-home\") pod \"2fb327ed-88b8-4f18-81b7-881fc4480618\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " Oct 30 17:37:17.591921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.591826 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-tmp-dir\") pod \"2fb327ed-88b8-4f18-81b7-881fc4480618\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " Oct 30 17:37:17.591921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.591856 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-model-cache\") pod \"2fb327ed-88b8-4f18-81b7-881fc4480618\" (UID: \"2fb327ed-88b8-4f18-81b7-881fc4480618\") " Oct 30 17:37:17.595467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.592546 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-home" pod "2fb327ed-88b8-4f18-81b7-881fc4480618" (UID: "2fb327ed-88b8-4f18-81b7-881fc4480618"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:17.595467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.592566 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-model-cache" pod "2fb327ed-88b8-4f18-81b7-881fc4480618" (UID: "2fb327ed-88b8-4f18-81b7-881fc4480618"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:17.595467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.592746 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-tmp-dir" pod "2fb327ed-88b8-4f18-81b7-881fc4480618" (UID: "2fb327ed-88b8-4f18-81b7-881fc4480618"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:17.595731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.595629 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb327ed-88b8-4f18-81b7-881fc4480618-kube-api-access-4h64p" pod "2fb327ed-88b8-4f18-81b7-881fc4480618" (UID: "2fb327ed-88b8-4f18-81b7-881fc4480618"). InnerVolumeSpecName "kube-api-access-4h64p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:17.595793 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.595730 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb327ed-88b8-4f18-81b7-881fc4480618-tls-certs" pod "2fb327ed-88b8-4f18-81b7-881fc4480618" (UID: "2fb327ed-88b8-4f18-81b7-881fc4480618"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:17.597248 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.597176 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-dshm" pod "2fb327ed-88b8-4f18-81b7-881fc4480618" (UID: "2fb327ed-88b8-4f18-81b7-881fc4480618"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:17.693869 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.693835 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:17.693869 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.693866 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:17.693869 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.693875 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fb327ed-88b8-4f18-81b7-881fc4480618-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:17.694137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.693883 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:17.694137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.693892 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h64p\" (UniqueName: \"kubernetes.io/projected/2fb327ed-88b8-4f18-81b7-881fc4480618-kube-api-access-4h64p\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:17.694436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:17.694133 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fb327ed-88b8-4f18-81b7-881fc4480618-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:18.436289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:18.436251 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" Oct 30 17:37:18.445523 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:18.445479 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:37:18.495824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:18.495790 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z"] Oct 30 17:37:18.504360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:18.504324 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-86f50a/distinct-2-w5-86f50a-kserve-d8b7dcc59-f4p8z"] Oct 30 17:37:18.737500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:18.737409 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:18.893971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:18.893176 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:19.056855 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:19.051039 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:19.706715 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:19.704352 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:20.377043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.376425 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr"] Oct 30 17:37:20.377043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.376672 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" podUID="606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a" containerName="maas-api" containerID="cri-o://683d84eaf6a8a25c30f7a1846905a59cc8cdfe339a2f71bd05ff641a0fd9bdd1" gracePeriod=30 Oct 30 17:37:20.457107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.457054 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:20.496213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.496178 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" path="/var/lib/kubelet/pods/2fb327ed-88b8-4f18-81b7-881fc4480618/volumes" Oct 30 17:37:20.833950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.831452 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 17:37:20.891096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.891060 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-openshift-service-ca\") pod \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " Oct 30 17:37:20.891306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.891134 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-maas-api-tls\") pod \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " Oct 30 17:37:20.891306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.891173 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-kube-api-access-fm27h\" (UniqueName: \"kubernetes.io/projected/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-kube-api-access-fm27h\") pod \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " Oct 30 17:37:20.891306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.891260 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-metrics-certs\") pod \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\" (UID: \"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a\") " Oct 30 17:37:20.891535 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.891508 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-openshift-service-ca" pod "606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a" (UID: "606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:20.892000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.891966 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:20.893797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.893773 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-metrics-certs" pod "606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a" (UID: "606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:20.897043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.897018 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-maas-api-tls" pod "606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a" (UID: "606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:20.902047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.902023 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-kube-api-access-fm27h" pod "606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a" (UID: "606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a"). InnerVolumeSpecName "kube-api-access-fm27h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:20.993708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.993663 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:20.993708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.993711 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm27h\" (UniqueName: \"kubernetes.io/projected/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-kube-api-access-fm27h\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:20.993951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:20.993733 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:21.027428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.027324 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:21.206927 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.202429 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:21.229258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.229049 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" podUID="666617d1-e514-4b17-ae65-1da9dfeea316" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:37:21.229648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.229623 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" podUID="14d8d98a-d3b2-4c79-9ebe-d9f817498f61" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:37:21.471469 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.471383 4828 generic.go:356] "Generic (PLEG): container finished" podID="606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a" containerID="683d84eaf6a8a25c30f7a1846905a59cc8cdfe339a2f71bd05ff641a0fd9bdd1" exitCode=0 Oct 30 17:37:21.471469 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.471427 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" event={"ID":"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a","Type":"ContainerDied","Data":"683d84eaf6a8a25c30f7a1846905a59cc8cdfe339a2f71bd05ff641a0fd9bdd1"} Oct 30 17:37:21.471673 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.471506 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" Oct 30 17:37:21.471673 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.471532 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr" event={"ID":"606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a","Type":"ContainerDied","Data":"a4e7bc69a4c7838b990dc655b25ad909fab0c35b925990431cb73f691525460b"} Oct 30 17:37:21.471673 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.471562 4828 scope.go:122] "RemoveContainer" containerID="683d84eaf6a8a25c30f7a1846905a59cc8cdfe339a2f71bd05ff641a0fd9bdd1" Oct 30 17:37:21.484921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.484876 4828 scope.go:122] "RemoveContainer" containerID="683d84eaf6a8a25c30f7a1846905a59cc8cdfe339a2f71bd05ff641a0fd9bdd1" Oct 30 17:37:21.485279 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:37:21.485256 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683d84eaf6a8a25c30f7a1846905a59cc8cdfe339a2f71bd05ff641a0fd9bdd1\": container with ID starting with 683d84eaf6a8a25c30f7a1846905a59cc8cdfe339a2f71bd05ff641a0fd9bdd1 not found: ID does not exist" containerID="683d84eaf6a8a25c30f7a1846905a59cc8cdfe339a2f71bd05ff641a0fd9bdd1" Oct 30 17:37:21.485350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.485288 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683d84eaf6a8a25c30f7a1846905a59cc8cdfe339a2f71bd05ff641a0fd9bdd1"} err="failed to get container status \"683d84eaf6a8a25c30f7a1846905a59cc8cdfe339a2f71bd05ff641a0fd9bdd1\": rpc error: code = NotFound desc = could not find container \"683d84eaf6a8a25c30f7a1846905a59cc8cdfe339a2f71bd05ff641a0fd9bdd1\": container with ID starting with 683d84eaf6a8a25c30f7a1846905a59cc8cdfe339a2f71bd05ff641a0fd9bdd1 not found: ID does not exist" Oct 30 17:37:21.530243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.530205 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr"] Oct 30 17:37:21.542838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.542803 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-b80d83-55476d78cb-dzsqr"] Oct 30 17:37:21.793668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:21.793638 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:22.493472 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:22.493433 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a" path="/var/lib/kubelet/pods/606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a/volumes" Oct 30 17:37:22.823780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:22.823743 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h"] Oct 30 17:37:22.824066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:22.824012 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" podUID="66bda2af-c577-42eb-bfdf-1b63d8433886" containerName="maas-api" containerID="cri-o://f773a2670898810e407ad7c8ee4cfe382c504d42eff177727a06f5d7f5034c92" gracePeriod=30 Oct 30 17:37:22.947237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:22.947208 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:23.155951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.155093 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 17:37:23.224131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.224096 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66bda2af-c577-42eb-bfdf-1b63d8433886-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bda2af-c577-42eb-bfdf-1b63d8433886-openshift-service-ca\") pod \"66bda2af-c577-42eb-bfdf-1b63d8433886\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " Oct 30 17:37:23.224358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.224263 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-maas-api-tls\") pod \"66bda2af-c577-42eb-bfdf-1b63d8433886\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " Oct 30 17:37:23.224358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.224290 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66bda2af-c577-42eb-bfdf-1b63d8433886-kube-api-access-97tg6\" (UniqueName: \"kubernetes.io/projected/66bda2af-c577-42eb-bfdf-1b63d8433886-kube-api-access-97tg6\") pod \"66bda2af-c577-42eb-bfdf-1b63d8433886\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " Oct 30 17:37:23.224358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.224333 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-metrics-certs\") pod \"66bda2af-c577-42eb-bfdf-1b63d8433886\" (UID: \"66bda2af-c577-42eb-bfdf-1b63d8433886\") " Oct 30 17:37:23.224563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.224534 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bda2af-c577-42eb-bfdf-1b63d8433886-openshift-service-ca" pod "66bda2af-c577-42eb-bfdf-1b63d8433886" (UID: "66bda2af-c577-42eb-bfdf-1b63d8433886"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:23.224789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.224768 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66bda2af-c577-42eb-bfdf-1b63d8433886-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:23.226527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.226501 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-maas-api-tls" pod "66bda2af-c577-42eb-bfdf-1b63d8433886" (UID: "66bda2af-c577-42eb-bfdf-1b63d8433886"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:23.226527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.226512 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bda2af-c577-42eb-bfdf-1b63d8433886-kube-api-access-97tg6" pod "66bda2af-c577-42eb-bfdf-1b63d8433886" (UID: "66bda2af-c577-42eb-bfdf-1b63d8433886"). InnerVolumeSpecName "kube-api-access-97tg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:23.226708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.226652 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-metrics-certs" pod "66bda2af-c577-42eb-bfdf-1b63d8433886" (UID: "66bda2af-c577-42eb-bfdf-1b63d8433886"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:23.326139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.326104 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:23.326139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.326135 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66bda2af-c577-42eb-bfdf-1b63d8433886-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:23.326139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.326144 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97tg6\" (UniqueName: \"kubernetes.io/projected/66bda2af-c577-42eb-bfdf-1b63d8433886-kube-api-access-97tg6\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:23.472340 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.471225 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv"] Oct 30 17:37:23.472340 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.471481 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" podUID="264f7990-6cc2-4018-a1c5-cc76ccfb5338" containerName="maas-api" containerID="cri-o://d201b3ffcb636acb0b884de8024cbc3357a050e852623ce5ba1c4ecdc95077c7" gracePeriod=30 Oct 30 17:37:23.505924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.499464 4828 generic.go:356] "Generic (PLEG): container finished" podID="66bda2af-c577-42eb-bfdf-1b63d8433886" containerID="f773a2670898810e407ad7c8ee4cfe382c504d42eff177727a06f5d7f5034c92" exitCode=0 Oct 30 17:37:23.505924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.499687 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" event={"ID":"66bda2af-c577-42eb-bfdf-1b63d8433886","Type":"ContainerDied","Data":"f773a2670898810e407ad7c8ee4cfe382c504d42eff177727a06f5d7f5034c92"} Oct 30 17:37:23.505924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.499740 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" event={"ID":"66bda2af-c577-42eb-bfdf-1b63d8433886","Type":"ContainerDied","Data":"e464b906497cd20708afd5485ccf63b5dfc888fac347aaa71b852fbe01da6f35"} Oct 30 17:37:23.505924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.499771 4828 scope.go:122] "RemoveContainer" containerID="f773a2670898810e407ad7c8ee4cfe382c504d42eff177727a06f5d7f5034c92" Oct 30 17:37:23.505924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.499888 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h" Oct 30 17:37:23.580459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.580421 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h"] Oct 30 17:37:23.586785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.586752 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b4f406-5cdcf6748-c8l9h"] Oct 30 17:37:23.588819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.588791 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:23.590563 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.590490 4828 scope.go:122] "RemoveContainer" containerID="f773a2670898810e407ad7c8ee4cfe382c504d42eff177727a06f5d7f5034c92" Oct 30 17:37:23.590978 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:37:23.590951 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f773a2670898810e407ad7c8ee4cfe382c504d42eff177727a06f5d7f5034c92\": container with ID starting with f773a2670898810e407ad7c8ee4cfe382c504d42eff177727a06f5d7f5034c92 not found: ID does not exist" containerID="f773a2670898810e407ad7c8ee4cfe382c504d42eff177727a06f5d7f5034c92" Oct 30 17:37:23.591077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.590992 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f773a2670898810e407ad7c8ee4cfe382c504d42eff177727a06f5d7f5034c92"} err="failed to get container status \"f773a2670898810e407ad7c8ee4cfe382c504d42eff177727a06f5d7f5034c92\": rpc error: code = NotFound desc = could not find container \"f773a2670898810e407ad7c8ee4cfe382c504d42eff177727a06f5d7f5034c92\": container with ID starting with f773a2670898810e407ad7c8ee4cfe382c504d42eff177727a06f5d7f5034c92 not found: ID does not exist" Oct 30 17:37:23.668379 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.668349 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg"] Oct 30 17:37:23.668798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.668769 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" podUID="dd6e82e3-8647-478a-b1d9-9be624746f21" containerName="payload-processing" containerID="cri-o://6bada3d2b00e31ad7e9543692ba46d5d175887c8f17264dd3e9930d27efa3bbb" gracePeriod=30 Oct 30 17:37:23.709002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.708962 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk"] Oct 30 17:37:23.709273 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.709209 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" podUID="52e9873d-a365-4097-af99-34bca8251f98" containerName="payload-pre-processing" containerID="cri-o://54ed652033ea9881424fc2f935d202c3d5582b128a7793a395737ba0df4752be" gracePeriod=30 Oct 30 17:37:23.755939 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.755867 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:23.835405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.835379 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:37:23.922351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.921664 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:23.941774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.936384 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264f7990-6cc2-4018-a1c5-cc76ccfb5338-kube-api-access-xjghp\" (UniqueName: \"kubernetes.io/projected/264f7990-6cc2-4018-a1c5-cc76ccfb5338-kube-api-access-xjghp\") pod \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " Oct 30 17:37:23.941774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.936472 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-metrics-certs\" (UniqueName: \"kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-metrics-certs\") pod \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " Oct 30 17:37:23.941774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.936558 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/264f7990-6cc2-4018-a1c5-cc76ccfb5338-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264f7990-6cc2-4018-a1c5-cc76ccfb5338-openshift-service-ca\") pod \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " Oct 30 17:37:23.941774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.936706 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-maas-api-tls\") pod \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\" (UID: \"264f7990-6cc2-4018-a1c5-cc76ccfb5338\") " Oct 30 17:37:23.941774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.938325 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264f7990-6cc2-4018-a1c5-cc76ccfb5338-openshift-service-ca" pod "264f7990-6cc2-4018-a1c5-cc76ccfb5338" (UID: "264f7990-6cc2-4018-a1c5-cc76ccfb5338"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:23.947017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.946972 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-metrics-certs" pod "264f7990-6cc2-4018-a1c5-cc76ccfb5338" (UID: "264f7990-6cc2-4018-a1c5-cc76ccfb5338"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:23.950212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.950175 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264f7990-6cc2-4018-a1c5-cc76ccfb5338-kube-api-access-xjghp" pod "264f7990-6cc2-4018-a1c5-cc76ccfb5338" (UID: "264f7990-6cc2-4018-a1c5-cc76ccfb5338"). InnerVolumeSpecName "kube-api-access-xjghp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:23.951088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:23.951059 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-maas-api-tls" pod "264f7990-6cc2-4018-a1c5-cc76ccfb5338" (UID: "264f7990-6cc2-4018-a1c5-cc76ccfb5338"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:24.051923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.050978 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjghp\" (UniqueName: \"kubernetes.io/projected/264f7990-6cc2-4018-a1c5-cc76ccfb5338-kube-api-access-xjghp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:24.051923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.051015 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:24.051923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.051031 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/264f7990-6cc2-4018-a1c5-cc76ccfb5338-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:24.051923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.051046 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/264f7990-6cc2-4018-a1c5-cc76ccfb5338-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:24.153820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.153784 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" Oct 30 17:37:24.165351 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.165325 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" Oct 30 17:37:24.253783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.253744 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6e82e3-8647-478a-b1d9-9be624746f21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6e82e3-8647-478a-b1d9-9be624746f21-plugins-config-volume\") pod \"dd6e82e3-8647-478a-b1d9-9be624746f21\" (UID: \"dd6e82e3-8647-478a-b1d9-9be624746f21\") " Oct 30 17:37:24.254001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.253815 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e9873d-a365-4097-af99-34bca8251f98-kube-api-access-xhc42\" (UniqueName: \"kubernetes.io/projected/52e9873d-a365-4097-af99-34bca8251f98-kube-api-access-xhc42\") pod \"52e9873d-a365-4097-af99-34bca8251f98\" (UID: \"52e9873d-a365-4097-af99-34bca8251f98\") " Oct 30 17:37:24.254001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.253847 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6e82e3-8647-478a-b1d9-9be624746f21-kube-api-access-mmnvs\" (UniqueName: \"kubernetes.io/projected/dd6e82e3-8647-478a-b1d9-9be624746f21-kube-api-access-mmnvs\") pod \"dd6e82e3-8647-478a-b1d9-9be624746f21\" (UID: \"dd6e82e3-8647-478a-b1d9-9be624746f21\") " Oct 30 17:37:24.254001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.253965 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52e9873d-a365-4097-af99-34bca8251f98-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e9873d-a365-4097-af99-34bca8251f98-plugins-config-volume\") pod \"52e9873d-a365-4097-af99-34bca8251f98\" (UID: \"52e9873d-a365-4097-af99-34bca8251f98\") " Oct 30 17:37:24.255722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.254419 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e9873d-a365-4097-af99-34bca8251f98-plugins-config-volume" pod "52e9873d-a365-4097-af99-34bca8251f98" (UID: "52e9873d-a365-4097-af99-34bca8251f98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:24.255722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.254769 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6e82e3-8647-478a-b1d9-9be624746f21-plugins-config-volume" pod "dd6e82e3-8647-478a-b1d9-9be624746f21" (UID: "dd6e82e3-8647-478a-b1d9-9be624746f21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:24.256294 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.256268 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6e82e3-8647-478a-b1d9-9be624746f21-kube-api-access-mmnvs" pod "dd6e82e3-8647-478a-b1d9-9be624746f21" (UID: "dd6e82e3-8647-478a-b1d9-9be624746f21"). InnerVolumeSpecName "kube-api-access-mmnvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:24.256448 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.256429 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e9873d-a365-4097-af99-34bca8251f98-kube-api-access-xhc42" pod "52e9873d-a365-4097-af99-34bca8251f98" (UID: "52e9873d-a365-4097-af99-34bca8251f98"). InnerVolumeSpecName "kube-api-access-xhc42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:24.355721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.355632 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd6e82e3-8647-478a-b1d9-9be624746f21-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:24.355721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.355669 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhc42\" (UniqueName: \"kubernetes.io/projected/52e9873d-a365-4097-af99-34bca8251f98-kube-api-access-xhc42\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:24.355721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.355679 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmnvs\" (UniqueName: \"kubernetes.io/projected/dd6e82e3-8647-478a-b1d9-9be624746f21-kube-api-access-mmnvs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:24.355721 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.355688 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52e9873d-a365-4097-af99-34bca8251f98-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:24.407343 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.407305 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:24.492040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.491994 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66bda2af-c577-42eb-bfdf-1b63d8433886" path="/var/lib/kubelet/pods/66bda2af-c577-42eb-bfdf-1b63d8433886/volumes" Oct 30 17:37:24.548452 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.548423 4828 generic.go:356] "Generic (PLEG): container finished" podID="52e9873d-a365-4097-af99-34bca8251f98" containerID="54ed652033ea9881424fc2f935d202c3d5582b128a7793a395737ba0df4752be" exitCode=0 Oct 30 17:37:24.548851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.548482 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" Oct 30 17:37:24.548851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.548506 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" event={"ID":"52e9873d-a365-4097-af99-34bca8251f98","Type":"ContainerDied","Data":"54ed652033ea9881424fc2f935d202c3d5582b128a7793a395737ba0df4752be"} Oct 30 17:37:24.548851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.548569 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk" event={"ID":"52e9873d-a365-4097-af99-34bca8251f98","Type":"ContainerDied","Data":"8e33224c75172aa14a08e2d5f91577449ed4e20e51f84ce1abb35439c08bc6dd"} Oct 30 17:37:24.548851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.548604 4828 scope.go:122] "RemoveContainer" containerID="54ed652033ea9881424fc2f935d202c3d5582b128a7793a395737ba0df4752be" Oct 30 17:37:24.557294 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.557095 4828 generic.go:356] "Generic (PLEG): container finished" podID="264f7990-6cc2-4018-a1c5-cc76ccfb5338" containerID="d201b3ffcb636acb0b884de8024cbc3357a050e852623ce5ba1c4ecdc95077c7" exitCode=0 Oct 30 17:37:24.557395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.557303 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" event={"ID":"264f7990-6cc2-4018-a1c5-cc76ccfb5338","Type":"ContainerDied","Data":"d201b3ffcb636acb0b884de8024cbc3357a050e852623ce5ba1c4ecdc95077c7"} Oct 30 17:37:24.557449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.557391 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" event={"ID":"264f7990-6cc2-4018-a1c5-cc76ccfb5338","Type":"ContainerDied","Data":"9276dedd8be531f8a8e52d98d0291228c5981681b360ed373a042f7ac06848b1"} Oct 30 17:37:24.557831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.557510 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv" Oct 30 17:37:24.561220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.561194 4828 generic.go:356] "Generic (PLEG): container finished" podID="dd6e82e3-8647-478a-b1d9-9be624746f21" containerID="6bada3d2b00e31ad7e9543692ba46d5d175887c8f17264dd3e9930d27efa3bbb" exitCode=0 Oct 30 17:37:24.561418 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.561389 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" event={"ID":"dd6e82e3-8647-478a-b1d9-9be624746f21","Type":"ContainerDied","Data":"6bada3d2b00e31ad7e9543692ba46d5d175887c8f17264dd3e9930d27efa3bbb"} Oct 30 17:37:24.561496 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.561430 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" Oct 30 17:37:24.561496 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.561459 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg" event={"ID":"dd6e82e3-8647-478a-b1d9-9be624746f21","Type":"ContainerDied","Data":"306b81ec33993eb42f4f99a58a1eb3146be16ef72e1b3244afd3c95b24cfe611"} Oct 30 17:37:24.574569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.574540 4828 scope.go:122] "RemoveContainer" containerID="54ed652033ea9881424fc2f935d202c3d5582b128a7793a395737ba0df4752be" Oct 30 17:37:24.574960 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:37:24.574935 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ed652033ea9881424fc2f935d202c3d5582b128a7793a395737ba0df4752be\": container with ID starting with 54ed652033ea9881424fc2f935d202c3d5582b128a7793a395737ba0df4752be not found: ID does not exist" containerID="54ed652033ea9881424fc2f935d202c3d5582b128a7793a395737ba0df4752be" Oct 30 17:37:24.575038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.574969 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ed652033ea9881424fc2f935d202c3d5582b128a7793a395737ba0df4752be"} err="failed to get container status \"54ed652033ea9881424fc2f935d202c3d5582b128a7793a395737ba0df4752be\": rpc error: code = NotFound desc = could not find container \"54ed652033ea9881424fc2f935d202c3d5582b128a7793a395737ba0df4752be\": container with ID starting with 54ed652033ea9881424fc2f935d202c3d5582b128a7793a395737ba0df4752be not found: ID does not exist" Oct 30 17:37:24.575038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.574989 4828 scope.go:122] "RemoveContainer" containerID="d201b3ffcb636acb0b884de8024cbc3357a050e852623ce5ba1c4ecdc95077c7" Oct 30 17:37:24.579980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.579943 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:24.587815 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.587794 4828 scope.go:122] "RemoveContainer" containerID="d201b3ffcb636acb0b884de8024cbc3357a050e852623ce5ba1c4ecdc95077c7" Oct 30 17:37:24.588119 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:37:24.588099 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d201b3ffcb636acb0b884de8024cbc3357a050e852623ce5ba1c4ecdc95077c7\": container with ID starting with d201b3ffcb636acb0b884de8024cbc3357a050e852623ce5ba1c4ecdc95077c7 not found: ID does not exist" containerID="d201b3ffcb636acb0b884de8024cbc3357a050e852623ce5ba1c4ecdc95077c7" Oct 30 17:37:24.588202 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.588126 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d201b3ffcb636acb0b884de8024cbc3357a050e852623ce5ba1c4ecdc95077c7"} err="failed to get container status \"d201b3ffcb636acb0b884de8024cbc3357a050e852623ce5ba1c4ecdc95077c7\": rpc error: code = NotFound desc = could not find container \"d201b3ffcb636acb0b884de8024cbc3357a050e852623ce5ba1c4ecdc95077c7\": container with ID starting with d201b3ffcb636acb0b884de8024cbc3357a050e852623ce5ba1c4ecdc95077c7 not found: ID does not exist" Oct 30 17:37:24.588202 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.588141 4828 scope.go:122] "RemoveContainer" containerID="6bada3d2b00e31ad7e9543692ba46d5d175887c8f17264dd3e9930d27efa3bbb" Oct 30 17:37:24.599843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.599822 4828 scope.go:122] "RemoveContainer" containerID="6bada3d2b00e31ad7e9543692ba46d5d175887c8f17264dd3e9930d27efa3bbb" Oct 30 17:37:24.600205 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:37:24.600183 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bada3d2b00e31ad7e9543692ba46d5d175887c8f17264dd3e9930d27efa3bbb\": container with ID starting with 6bada3d2b00e31ad7e9543692ba46d5d175887c8f17264dd3e9930d27efa3bbb not found: ID does not exist" containerID="6bada3d2b00e31ad7e9543692ba46d5d175887c8f17264dd3e9930d27efa3bbb" Oct 30 17:37:24.600266 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.600215 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bada3d2b00e31ad7e9543692ba46d5d175887c8f17264dd3e9930d27efa3bbb"} err="failed to get container status \"6bada3d2b00e31ad7e9543692ba46d5d175887c8f17264dd3e9930d27efa3bbb\": rpc error: code = NotFound desc = could not find container \"6bada3d2b00e31ad7e9543692ba46d5d175887c8f17264dd3e9930d27efa3bbb\": container with ID starting with 6bada3d2b00e31ad7e9543692ba46d5d175887c8f17264dd3e9930d27efa3bbb not found: ID does not exist" Oct 30 17:37:24.611057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.610982 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv"] Oct 30 17:37:24.615226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.615195 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a949e7-7bb4777964-7tlhv"] Oct 30 17:37:24.655389 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.655359 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk"] Oct 30 17:37:24.659630 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.659602 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a949e7-65f96bd8cf-9pjsk"] Oct 30 17:37:24.696417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.696384 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg"] Oct 30 17:37:24.719842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:24.719802 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a949e7-87df6f6b8-zvbtg"] Oct 30 17:37:25.104263 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:25.104231 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:26.503666 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:26.503634 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="264f7990-6cc2-4018-a1c5-cc76ccfb5338" path="/var/lib/kubelet/pods/264f7990-6cc2-4018-a1c5-cc76ccfb5338/volumes" Oct 30 17:37:26.504096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:26.504080 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e9873d-a365-4097-af99-34bca8251f98" path="/var/lib/kubelet/pods/52e9873d-a365-4097-af99-34bca8251f98/volumes" Oct 30 17:37:26.504471 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:26.504458 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6e82e3-8647-478a-b1d9-9be624746f21" path="/var/lib/kubelet/pods/dd6e82e3-8647-478a-b1d9-9be624746f21/volumes" Oct 30 17:37:26.534790 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:26.534755 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:27.332167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:27.331319 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:27.512988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:27.505323 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:27.669979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:27.668893 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:28.478482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:28.478447 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:29.004950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:29.004913 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:29.216766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:29.216725 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:29.393743 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:29.393688 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:29.558028 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:29.557989 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll"] Oct 30 17:37:29.558614 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:29.558577 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" containerID="cri-o://0d85d09906cce7f74b3d058b6aaff289414b47a7ae9d2dda54f831f410203300" gracePeriod=30 Oct 30 17:37:29.608705 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:29.608658 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:29.781929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:29.781873 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:29.937646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:29.937602 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:30.905064 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:30.903148 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:31.213388 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:31.213298 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" podUID="14d8d98a-d3b2-4c79-9ebe-d9f817498f61" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:37:31.213388 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:31.213343 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" podUID="666617d1-e514-4b17-ae65-1da9dfeea316" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:37:32.029642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.029607 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:32.183165 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.183068 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:32.357763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.357633 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:32.541962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.538992 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:32.714477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.710671 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:32.822002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.821963 4828 generic.go:356] "Generic (PLEG): container finished" podID="666617d1-e514-4b17-ae65-1da9dfeea316" containerID="234edcfaa0fd28d71f3e68f93d2456a12083e270067bf98e5d652e0ca18c9588" exitCode=0 Oct 30 17:37:32.822157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.822028 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" event={"ID":"666617d1-e514-4b17-ae65-1da9dfeea316","Type":"ContainerDied","Data":"234edcfaa0fd28d71f3e68f93d2456a12083e270067bf98e5d652e0ca18c9588"} Oct 30 17:37:32.822157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.822088 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" event={"ID":"666617d1-e514-4b17-ae65-1da9dfeea316","Type":"ContainerDied","Data":"cbe24a0395b5b6f80d7348809915e02aa295422a857575b6b75dbface0571404"} Oct 30 17:37:32.822157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.822109 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbe24a0395b5b6f80d7348809915e02aa295422a857575b6b75dbface0571404" Oct 30 17:37:32.823846 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.823826 4828 generic.go:356] "Generic (PLEG): container finished" podID="14d8d98a-d3b2-4c79-9ebe-d9f817498f61" containerID="336b6335ef8493e9436208e409aca31f68e807b9e164599ede7bc7e7e1bf06aa" exitCode=0 Oct 30 17:37:32.823991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.823895 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" event={"ID":"14d8d98a-d3b2-4c79-9ebe-d9f817498f61","Type":"ContainerDied","Data":"336b6335ef8493e9436208e409aca31f68e807b9e164599ede7bc7e7e1bf06aa"} Oct 30 17:37:32.840433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.840406 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" Oct 30 17:37:32.922030 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.921886 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:32.943814 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.943770 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666617d1-e514-4b17-ae65-1da9dfeea316-kube-api-access-m8cnh\" (UniqueName: \"kubernetes.io/projected/666617d1-e514-4b17-ae65-1da9dfeea316-kube-api-access-m8cnh\") pod \"666617d1-e514-4b17-ae65-1da9dfeea316\" (UID: \"666617d1-e514-4b17-ae65-1da9dfeea316\") " Oct 30 17:37:32.944707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.944684 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666617d1-e514-4b17-ae65-1da9dfeea316-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666617d1-e514-4b17-ae65-1da9dfeea316-plugins-config-volume\") pod \"666617d1-e514-4b17-ae65-1da9dfeea316\" (UID: \"666617d1-e514-4b17-ae65-1da9dfeea316\") " Oct 30 17:37:32.945811 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.945223 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666617d1-e514-4b17-ae65-1da9dfeea316-plugins-config-volume" pod "666617d1-e514-4b17-ae65-1da9dfeea316" (UID: "666617d1-e514-4b17-ae65-1da9dfeea316"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:32.946739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.946671 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666617d1-e514-4b17-ae65-1da9dfeea316-kube-api-access-m8cnh" pod "666617d1-e514-4b17-ae65-1da9dfeea316" (UID: "666617d1-e514-4b17-ae65-1da9dfeea316"). InnerVolumeSpecName "kube-api-access-m8cnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:32.968106 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:32.968033 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" Oct 30 17:37:33.046372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.046333 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-plugins-config-volume\") pod \"14d8d98a-d3b2-4c79-9ebe-d9f817498f61\" (UID: \"14d8d98a-d3b2-4c79-9ebe-d9f817498f61\") " Oct 30 17:37:33.046967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.046401 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-kube-api-access-99fmh\" (UniqueName: \"kubernetes.io/projected/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-kube-api-access-99fmh\") pod \"14d8d98a-d3b2-4c79-9ebe-d9f817498f61\" (UID: \"14d8d98a-d3b2-4c79-9ebe-d9f817498f61\") " Oct 30 17:37:33.047098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.046996 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8cnh\" (UniqueName: \"kubernetes.io/projected/666617d1-e514-4b17-ae65-1da9dfeea316-kube-api-access-m8cnh\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:33.047098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.047023 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/666617d1-e514-4b17-ae65-1da9dfeea316-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:33.047801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.047772 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-plugins-config-volume" pod "14d8d98a-d3b2-4c79-9ebe-d9f817498f61" (UID: "14d8d98a-d3b2-4c79-9ebe-d9f817498f61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:33.050095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.050063 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-kube-api-access-99fmh" pod "14d8d98a-d3b2-4c79-9ebe-d9f817498f61" (UID: "14d8d98a-d3b2-4c79-9ebe-d9f817498f61"). InnerVolumeSpecName "kube-api-access-99fmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:33.148432 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.148395 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:33.148432 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.148428 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99fmh\" (UniqueName: \"kubernetes.io/projected/14d8d98a-d3b2-4c79-9ebe-d9f817498f61-kube-api-access-99fmh\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:33.832799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.832762 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" Oct 30 17:37:33.833083 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.832790 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh" event={"ID":"14d8d98a-d3b2-4c79-9ebe-d9f817498f61","Type":"ContainerDied","Data":"1a51d94c1f8e64dd25288db9323354aa68b7852115507ae481008a7ca332630b"} Oct 30 17:37:33.833083 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.832861 4828 scope.go:122] "RemoveContainer" containerID="336b6335ef8493e9436208e409aca31f68e807b9e164599ede7bc7e7e1bf06aa" Oct 30 17:37:33.833208 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.833096 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf" Oct 30 17:37:33.897941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.893156 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf"] Oct 30 17:37:33.898506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.898465 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-86f50a-68fb5764f6-6mqqf"] Oct 30 17:37:33.935011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.934974 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh"] Oct 30 17:37:33.942489 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:33.942458 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-86f50a-75c9744799-t8dbh"] Oct 30 17:37:34.493687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:34.493647 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d8d98a-d3b2-4c79-9ebe-d9f817498f61" path="/var/lib/kubelet/pods/14d8d98a-d3b2-4c79-9ebe-d9f817498f61/volumes" Oct 30 17:37:34.494304 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:34.494284 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666617d1-e514-4b17-ae65-1da9dfeea316" path="/var/lib/kubelet/pods/666617d1-e514-4b17-ae65-1da9dfeea316/volumes" Oct 30 17:37:34.863602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:34.863560 4828 generic.go:356] "Generic (PLEG): container finished" podID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerID="0d85d09906cce7f74b3d058b6aaff289414b47a7ae9d2dda54f831f410203300" exitCode=0 Oct 30 17:37:34.863794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:34.863699 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" event={"ID":"e2a75163-e6e5-4eaa-9197-a0e16de86f19","Type":"ContainerDied","Data":"0d85d09906cce7f74b3d058b6aaff289414b47a7ae9d2dda54f831f410203300"} Oct 30 17:37:34.863864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:34.863802 4828 scope.go:122] "RemoveContainer" containerID="07db4d7706f33854aa2205387dbb67ba093f9bcc7f3e3a624319a67c9d0471ec" Oct 30 17:37:34.956259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:34.956231 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 17:37:35.083698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.083613 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-token\" (UniqueName: \"kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-token\") pod \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " Oct 30 17:37:35.083864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.083737 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-socket\") pod \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " Oct 30 17:37:35.083864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.083774 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-kube-api-access-fnmmw\" (UniqueName: \"kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-kube-api-access-fnmmw\") pod \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " Oct 30 17:37:35.083864 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.083807 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istiod-ca-cert\") pod \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " Oct 30 17:37:35.084093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.083963 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-data\") pod \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " Oct 30 17:37:35.084093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.084010 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-certs\") pod \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " Oct 30 17:37:35.084093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.084049 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-envoy\") pod \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " Oct 30 17:37:35.084269 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.084109 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-podinfo\") pod \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " Oct 30 17:37:35.084269 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.084146 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-credential-socket\") pod \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\" (UID: \"e2a75163-e6e5-4eaa-9197-a0e16de86f19\") " Oct 30 17:37:35.084858 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.084102 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-socket" pod "e2a75163-e6e5-4eaa-9197-a0e16de86f19" (UID: "e2a75163-e6e5-4eaa-9197-a0e16de86f19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:35.085068 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.084350 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istiod-ca-cert" pod "e2a75163-e6e5-4eaa-9197-a0e16de86f19" (UID: "e2a75163-e6e5-4eaa-9197-a0e16de86f19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:35.085184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.084594 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-data" pod "e2a75163-e6e5-4eaa-9197-a0e16de86f19" (UID: "e2a75163-e6e5-4eaa-9197-a0e16de86f19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:35.085310 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.084761 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-certs" pod "e2a75163-e6e5-4eaa-9197-a0e16de86f19" (UID: "e2a75163-e6e5-4eaa-9197-a0e16de86f19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:35.085444 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.084833 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-credential-socket" pod "e2a75163-e6e5-4eaa-9197-a0e16de86f19" (UID: "e2a75163-e6e5-4eaa-9197-a0e16de86f19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:35.087074 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.087047 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-token" pod "e2a75163-e6e5-4eaa-9197-a0e16de86f19" (UID: "e2a75163-e6e5-4eaa-9197-a0e16de86f19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:35.087251 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.087224 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-envoy" pod "e2a75163-e6e5-4eaa-9197-a0e16de86f19" (UID: "e2a75163-e6e5-4eaa-9197-a0e16de86f19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:37:35.089029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.089002 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-kube-api-access-fnmmw" pod "e2a75163-e6e5-4eaa-9197-a0e16de86f19" (UID: "e2a75163-e6e5-4eaa-9197-a0e16de86f19"). InnerVolumeSpecName "kube-api-access-fnmmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:35.096119 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.096085 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-podinfo" pod "e2a75163-e6e5-4eaa-9197-a0e16de86f19" (UID: "e2a75163-e6e5-4eaa-9197-a0e16de86f19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:37:35.185964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.185885 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:35.185964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.185956 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:35.185964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.185969 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:35.186218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.185983 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:35.186218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.185998 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:35.186218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.186009 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:35.186218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.186021 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnmmw\" (UniqueName: \"kubernetes.io/projected/e2a75163-e6e5-4eaa-9197-a0e16de86f19-kube-api-access-fnmmw\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:35.186218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.186034 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:35.186218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.186047 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a75163-e6e5-4eaa-9197-a0e16de86f19-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:35.222407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.222365 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:35.377507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.377421 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:35.613370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.613325 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:35.878787 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.878747 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" Oct 30 17:37:35.879033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.878790 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll" event={"ID":"e2a75163-e6e5-4eaa-9197-a0e16de86f19","Type":"ContainerDied","Data":"0094ed4344b21270d28210d43197283aac3fda496eda16d3149fec231ecef089"} Oct 30 17:37:35.879033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.878850 4828 scope.go:122] "RemoveContainer" containerID="0d85d09906cce7f74b3d058b6aaff289414b47a7ae9d2dda54f831f410203300" Oct 30 17:37:35.969089 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.969053 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll"] Oct 30 17:37:35.980129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:35.974555 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b80d83-openshift-default-6bd4b6dc7c-hfdll"] Oct 30 17:37:36.491354 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:36.491323 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" path="/var/lib/kubelet/pods/e2a75163-e6e5-4eaa-9197-a0e16de86f19/volumes" Oct 30 17:37:36.803414 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:36.803374 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 17:37:36.803827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:36.803431 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 17:37:36.890914 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:36.890869 4828 generic.go:356] "Generic (PLEG): container finished" podID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerID="503d48ce2722fd90e8109e1a7907d3b4bd73fc0451a7875047a1961b6b97b6bb" exitCode=0 Oct 30 17:37:36.891094 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:36.890951 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" event={"ID":"a3660fa4-e79d-4624-a792-d60275ef4c43","Type":"ContainerDied","Data":"503d48ce2722fd90e8109e1a7907d3b4bd73fc0451a7875047a1961b6b97b6bb"} Oct 30 17:37:36.891393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:36.891379 4828 scope.go:122] "RemoveContainer" containerID="503d48ce2722fd90e8109e1a7907d3b4bd73fc0451a7875047a1961b6b97b6bb" Oct 30 17:37:36.893013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:36.892984 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:37:36.893152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:36.893069 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:37:36.893152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:36.893116 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:37:37.841498 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:37.841447 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:37.917659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:37.917625 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" event={"ID":"a3660fa4-e79d-4624-a792-d60275ef4c43","Type":"ContainerStarted","Data":"7c0b95184ad45c10af27ce58166e660281099b693d3078618d4e130fe761f864"} Oct 30 17:37:37.939961 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:37:37.939926 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:37:38.028587 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:38.028549 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:37:38.028941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:38.028888 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 17:37:38.029039 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:38.028965 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 17:37:39.028280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:39.028237 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 17:37:39.028715 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:39.028313 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 17:37:39.617056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:39.616514 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:40.029935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:40.028548 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 17:37:40.029935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:40.028613 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 17:37:41.028574 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:41.028528 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 17:37:41.028762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:41.028590 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 17:37:41.239262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:41.238491 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:42.028225 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:42.028187 4828 patch_prober.go:29] interesting pod/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body= Oct 30 17:37:42.028417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:42.028256 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" Oct 30 17:37:43.033782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:43.033745 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:37:43.034288 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:43.034207 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:37:43.034990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:43.034971 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:37:43.269149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:43.269112 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:44.915469 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:44.915214 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:46.072658 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:46.072582 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:46.239078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:46.239032 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:47.200410 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:47.200367 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:48.979120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:48.979021 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:50.522066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:50.522015 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:52.264599 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:52.264515 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:53.994275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:53.994236 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7"] Oct 30 17:37:53.994725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:53.994517 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" podUID="07e31526-445e-4541-8026-e134539da213" containerName="maas-api" containerID="cri-o://8c8cc74f33cfb7c233827b75d01d7961b691dd345022d5829f4d72a3b65bdb0e" gracePeriod=30 Oct 30 17:37:54.105403 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.105362 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:54.319618 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.319593 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 17:37:54.467287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.467241 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:54.500475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.500444 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e31526-445e-4541-8026-e134539da213-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07e31526-445e-4541-8026-e134539da213-openshift-service-ca\") pod \"07e31526-445e-4541-8026-e134539da213\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " Oct 30 17:37:54.500678 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.500487 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-metrics-certs\") pod \"07e31526-445e-4541-8026-e134539da213\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " Oct 30 17:37:54.500889 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.500866 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e31526-445e-4541-8026-e134539da213-openshift-service-ca" pod "07e31526-445e-4541-8026-e134539da213" (UID: "07e31526-445e-4541-8026-e134539da213"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:37:54.500984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.500921 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-maas-api-tls\") pod \"07e31526-445e-4541-8026-e134539da213\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " Oct 30 17:37:54.501045 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.500983 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e31526-445e-4541-8026-e134539da213-kube-api-access-7k9nf\" (UniqueName: \"kubernetes.io/projected/07e31526-445e-4541-8026-e134539da213-kube-api-access-7k9nf\") pod \"07e31526-445e-4541-8026-e134539da213\" (UID: \"07e31526-445e-4541-8026-e134539da213\") " Oct 30 17:37:54.501453 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.501399 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07e31526-445e-4541-8026-e134539da213-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:54.502835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.502812 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-metrics-certs" pod "07e31526-445e-4541-8026-e134539da213" (UID: "07e31526-445e-4541-8026-e134539da213"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:54.503081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.503061 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-maas-api-tls" pod "07e31526-445e-4541-8026-e134539da213" (UID: "07e31526-445e-4541-8026-e134539da213"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:37:54.503201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.503156 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e31526-445e-4541-8026-e134539da213-kube-api-access-7k9nf" pod "07e31526-445e-4541-8026-e134539da213" (UID: "07e31526-445e-4541-8026-e134539da213"). InnerVolumeSpecName "kube-api-access-7k9nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:37:54.602252 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.602209 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:54.602252 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.602246 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07e31526-445e-4541-8026-e134539da213-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:54.602473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.602259 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k9nf\" (UniqueName: \"kubernetes.io/projected/07e31526-445e-4541-8026-e134539da213-kube-api-access-7k9nf\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:37:54.624213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.624176 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:54.833555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:54.833462 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:55.068165 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:55.068129 4828 generic.go:356] "Generic (PLEG): container finished" podID="07e31526-445e-4541-8026-e134539da213" containerID="8c8cc74f33cfb7c233827b75d01d7961b691dd345022d5829f4d72a3b65bdb0e" exitCode=0 Oct 30 17:37:55.068660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:55.068173 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" Oct 30 17:37:55.068660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:55.068217 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" event={"ID":"07e31526-445e-4541-8026-e134539da213","Type":"ContainerDied","Data":"8c8cc74f33cfb7c233827b75d01d7961b691dd345022d5829f4d72a3b65bdb0e"} Oct 30 17:37:55.068660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:55.068273 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7" event={"ID":"07e31526-445e-4541-8026-e134539da213","Type":"ContainerDied","Data":"d69886c8f3358a0c2316c8f62e5e384c9fc98adf516363bfadd5ce09027fa1e8"} Oct 30 17:37:55.068660 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:55.068297 4828 scope.go:122] "RemoveContainer" containerID="8c8cc74f33cfb7c233827b75d01d7961b691dd345022d5829f4d72a3b65bdb0e" Oct 30 17:37:55.080910 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:55.080879 4828 scope.go:122] "RemoveContainer" containerID="8c8cc74f33cfb7c233827b75d01d7961b691dd345022d5829f4d72a3b65bdb0e" Oct 30 17:37:55.081217 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:37:55.081198 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8cc74f33cfb7c233827b75d01d7961b691dd345022d5829f4d72a3b65bdb0e\": container with ID starting with 8c8cc74f33cfb7c233827b75d01d7961b691dd345022d5829f4d72a3b65bdb0e not found: ID does not exist" containerID="8c8cc74f33cfb7c233827b75d01d7961b691dd345022d5829f4d72a3b65bdb0e" Oct 30 17:37:55.081291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:55.081223 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8cc74f33cfb7c233827b75d01d7961b691dd345022d5829f4d72a3b65bdb0e"} err="failed to get container status \"8c8cc74f33cfb7c233827b75d01d7961b691dd345022d5829f4d72a3b65bdb0e\": rpc error: code = NotFound desc = could not find container \"8c8cc74f33cfb7c233827b75d01d7961b691dd345022d5829f4d72a3b65bdb0e\": container with ID starting with 8c8cc74f33cfb7c233827b75d01d7961b691dd345022d5829f4d72a3b65bdb0e not found: ID does not exist" Oct 30 17:37:55.123607 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:55.123525 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7"] Oct 30 17:37:55.135708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:55.135669 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-86f50a-796cdd8cfc-rgxj7"] Oct 30 17:37:55.156821 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:55.156779 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:55.332022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:55.328404 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:55.490969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:55.490848 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:55.837886 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:55.837843 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:56.490766 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:56.490731 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e31526-445e-4541-8026-e134539da213" path="/var/lib/kubelet/pods/07e31526-445e-4541-8026-e134539da213/volumes" Oct 30 17:37:57.053937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:57.053823 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:37:58.846499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:37:58.846459 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:00.405344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:00.405309 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:00.558104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:00.558068 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:02.128781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:02.128741 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:02.329945 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:02.329882 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:02.568138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:02.568015 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:03.139716 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:03.139678 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:03.408662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:03.408580 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:03.570284 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:03.570243 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:03.699333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:03.699243 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k"] Oct 30 17:38:03.699566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:03.699535 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" containerID="cri-o://0881742fccc5d7c0fe444d64762aa399c1e52ff51c3a8909094225d1c935ed4e" gracePeriod=30 Oct 30 17:38:03.778385 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:03.778338 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:04.518079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:04.518034 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:06.249984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:06.249945 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:07.876760 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:07.876360 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:07.913497 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:38:07.913439 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:38:09.050970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.050570 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:38:09.123936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.123883 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-data\") pod \"c249be1c-0e0f-4380-917c-bb42dc4308c8\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " Oct 30 17:38:09.124140 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.123989 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-envoy\") pod \"c249be1c-0e0f-4380-917c-bb42dc4308c8\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " Oct 30 17:38:09.124140 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.124019 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-certs\") pod \"c249be1c-0e0f-4380-917c-bb42dc4308c8\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " Oct 30 17:38:09.124140 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.124116 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c249be1c-0e0f-4380-917c-bb42dc4308c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c249be1c-0e0f-4380-917c-bb42dc4308c8-istiod-ca-cert\") pod \"c249be1c-0e0f-4380-917c-bb42dc4308c8\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " Oct 30 17:38:09.124339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.124147 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-kube-api-access-pwsz7\" (UniqueName: \"kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-kube-api-access-pwsz7\") pod \"c249be1c-0e0f-4380-917c-bb42dc4308c8\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " Oct 30 17:38:09.124339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.124188 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-podinfo\") pod \"c249be1c-0e0f-4380-917c-bb42dc4308c8\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " Oct 30 17:38:09.124339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.124214 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-data" pod "c249be1c-0e0f-4380-917c-bb42dc4308c8" (UID: "c249be1c-0e0f-4380-917c-bb42dc4308c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:09.124339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.124239 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-credential-socket\") pod \"c249be1c-0e0f-4380-917c-bb42dc4308c8\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " Oct 30 17:38:09.124339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.124290 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-socket\") pod \"c249be1c-0e0f-4380-917c-bb42dc4308c8\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " Oct 30 17:38:09.124633 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.124353 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-token\" (UniqueName: \"kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-token\") pod \"c249be1c-0e0f-4380-917c-bb42dc4308c8\" (UID: \"c249be1c-0e0f-4380-917c-bb42dc4308c8\") " Oct 30 17:38:09.124782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.124748 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-socket" pod "c249be1c-0e0f-4380-917c-bb42dc4308c8" (UID: "c249be1c-0e0f-4380-917c-bb42dc4308c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:09.124942 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.124812 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:09.125283 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.125121 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-credential-socket" pod "c249be1c-0e0f-4380-917c-bb42dc4308c8" (UID: "c249be1c-0e0f-4380-917c-bb42dc4308c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:09.125283 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.125260 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c249be1c-0e0f-4380-917c-bb42dc4308c8-istiod-ca-cert" pod "c249be1c-0e0f-4380-917c-bb42dc4308c8" (UID: "c249be1c-0e0f-4380-917c-bb42dc4308c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:09.125627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.125606 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-certs" pod "c249be1c-0e0f-4380-917c-bb42dc4308c8" (UID: "c249be1c-0e0f-4380-917c-bb42dc4308c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:09.128087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.128065 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-envoy" pod "c249be1c-0e0f-4380-917c-bb42dc4308c8" (UID: "c249be1c-0e0f-4380-917c-bb42dc4308c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:09.128489 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.128466 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-kube-api-access-pwsz7" pod "c249be1c-0e0f-4380-917c-bb42dc4308c8" (UID: "c249be1c-0e0f-4380-917c-bb42dc4308c8"). InnerVolumeSpecName "kube-api-access-pwsz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:09.130198 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.130175 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-podinfo" pod "c249be1c-0e0f-4380-917c-bb42dc4308c8" (UID: "c249be1c-0e0f-4380-917c-bb42dc4308c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:09.131610 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.131292 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-token" pod "c249be1c-0e0f-4380-917c-bb42dc4308c8" (UID: "c249be1c-0e0f-4380-917c-bb42dc4308c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:09.170723 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.170685 4828 generic.go:356] "Generic (PLEG): container finished" podID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerID="0881742fccc5d7c0fe444d64762aa399c1e52ff51c3a8909094225d1c935ed4e" exitCode=0 Oct 30 17:38:09.170949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.170752 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" Oct 30 17:38:09.170949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.170765 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" event={"ID":"c249be1c-0e0f-4380-917c-bb42dc4308c8","Type":"ContainerDied","Data":"0881742fccc5d7c0fe444d64762aa399c1e52ff51c3a8909094225d1c935ed4e"} Oct 30 17:38:09.170949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.170866 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k" event={"ID":"c249be1c-0e0f-4380-917c-bb42dc4308c8","Type":"ContainerDied","Data":"284ee22ce46855300478fa0b3efade054f51d09d5e13a0cab30235c6eefa85d7"} Oct 30 17:38:09.170949 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.170928 4828 scope.go:122] "RemoveContainer" containerID="0881742fccc5d7c0fe444d64762aa399c1e52ff51c3a8909094225d1c935ed4e" Oct 30 17:38:09.184043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.184013 4828 scope.go:122] "RemoveContainer" containerID="ebcf70a9fc3860f0899fc2c6231c5abe4de1bf6de2371db824b856ce1a70a097" Oct 30 17:38:09.194611 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.194586 4828 scope.go:122] "RemoveContainer" containerID="0881742fccc5d7c0fe444d64762aa399c1e52ff51c3a8909094225d1c935ed4e" Oct 30 17:38:09.194960 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:38:09.194935 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0881742fccc5d7c0fe444d64762aa399c1e52ff51c3a8909094225d1c935ed4e\": container with ID starting with 0881742fccc5d7c0fe444d64762aa399c1e52ff51c3a8909094225d1c935ed4e not found: ID does not exist" containerID="0881742fccc5d7c0fe444d64762aa399c1e52ff51c3a8909094225d1c935ed4e" Oct 30 17:38:09.195066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.194974 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0881742fccc5d7c0fe444d64762aa399c1e52ff51c3a8909094225d1c935ed4e"} err="failed to get container status \"0881742fccc5d7c0fe444d64762aa399c1e52ff51c3a8909094225d1c935ed4e\": rpc error: code = NotFound desc = could not find container \"0881742fccc5d7c0fe444d64762aa399c1e52ff51c3a8909094225d1c935ed4e\": container with ID starting with 0881742fccc5d7c0fe444d64762aa399c1e52ff51c3a8909094225d1c935ed4e not found: ID does not exist" Oct 30 17:38:09.195066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.194999 4828 scope.go:122] "RemoveContainer" containerID="ebcf70a9fc3860f0899fc2c6231c5abe4de1bf6de2371db824b856ce1a70a097" Oct 30 17:38:09.195355 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:38:09.195332 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebcf70a9fc3860f0899fc2c6231c5abe4de1bf6de2371db824b856ce1a70a097\": container with ID starting with ebcf70a9fc3860f0899fc2c6231c5abe4de1bf6de2371db824b856ce1a70a097 not found: ID does not exist" containerID="ebcf70a9fc3860f0899fc2c6231c5abe4de1bf6de2371db824b856ce1a70a097" Oct 30 17:38:09.195438 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.195364 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebcf70a9fc3860f0899fc2c6231c5abe4de1bf6de2371db824b856ce1a70a097"} err="failed to get container status \"ebcf70a9fc3860f0899fc2c6231c5abe4de1bf6de2371db824b856ce1a70a097\": rpc error: code = NotFound desc = could not find container \"ebcf70a9fc3860f0899fc2c6231c5abe4de1bf6de2371db824b856ce1a70a097\": container with ID starting with ebcf70a9fc3860f0899fc2c6231c5abe4de1bf6de2371db824b856ce1a70a097 not found: ID does not exist" Oct 30 17:38:09.225923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.225833 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:09.225923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.225870 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:09.225923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.225884 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c249be1c-0e0f-4380-917c-bb42dc4308c8-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:09.225923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.225914 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwsz7\" (UniqueName: \"kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-kube-api-access-pwsz7\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:09.225923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.225930 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:09.226341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.225947 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:09.226341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.225960 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c249be1c-0e0f-4380-917c-bb42dc4308c8-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:09.226341 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.225974 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c249be1c-0e0f-4380-917c-bb42dc4308c8-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:09.233967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.229978 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k"] Oct 30 17:38:09.233967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.232701 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-031223-openshift-default-58cc9f8755-n4l2k"] Oct 30 17:38:09.798919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:09.797218 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:10.492606 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:10.492566 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" path="/var/lib/kubelet/pods/c249be1c-0e0f-4380-917c-bb42dc4308c8/volumes" Oct 30 17:38:11.974538 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:11.974495 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:12.953159 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:12.953120 4828 scope.go:122] "RemoveContainer" containerID="234edcfaa0fd28d71f3e68f93d2456a12083e270067bf98e5d652e0ca18c9588" Oct 30 17:38:12.965550 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:12.965525 4828 scope.go:122] "RemoveContainer" containerID="43568ab0be19ea1ac8aaacc7a72253e196283fc6c64f4248d7a18c7964f7d2e2" Oct 30 17:38:13.634511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:13.634473 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:14.357774 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:14.357729 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7"] Oct 30 17:38:14.392583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:14.392537 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75"] Oct 30 17:38:14.554818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:14.554772 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:14.750437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:14.750358 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:15.225037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:15.217333 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:16.515400 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:16.515363 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84"] Oct 30 17:38:16.515959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:16.515595 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" podUID="93b9082e-ac02-4490-a9aa-d805d7f9d737" containerName="maas-api" containerID="cri-o://b90658b778c40e407d8f54f499960045d86c839bd376f8e118a97a2d1f88adb1" gracePeriod=30 Oct 30 17:38:16.624590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:16.624549 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:16.827271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:16.827246 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:38:16.928921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:16.928561 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-maas-api-tls\") pod \"93b9082e-ac02-4490-a9aa-d805d7f9d737\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " Oct 30 17:38:16.928921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:16.928791 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93b9082e-ac02-4490-a9aa-d805d7f9d737-kube-api-access-2g6kt\" (UniqueName: \"kubernetes.io/projected/93b9082e-ac02-4490-a9aa-d805d7f9d737-kube-api-access-2g6kt\") pod \"93b9082e-ac02-4490-a9aa-d805d7f9d737\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " Oct 30 17:38:16.928921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:16.928851 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93b9082e-ac02-4490-a9aa-d805d7f9d737-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b9082e-ac02-4490-a9aa-d805d7f9d737-openshift-service-ca\") pod \"93b9082e-ac02-4490-a9aa-d805d7f9d737\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " Oct 30 17:38:16.929256 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:16.928949 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-metrics-certs\") pod \"93b9082e-ac02-4490-a9aa-d805d7f9d737\" (UID: \"93b9082e-ac02-4490-a9aa-d805d7f9d737\") " Oct 30 17:38:16.949571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:16.949507 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-maas-api-tls" pod "93b9082e-ac02-4490-a9aa-d805d7f9d737" (UID: "93b9082e-ac02-4490-a9aa-d805d7f9d737"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:38:16.950195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:16.950097 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93b9082e-ac02-4490-a9aa-d805d7f9d737-openshift-service-ca" pod "93b9082e-ac02-4490-a9aa-d805d7f9d737" (UID: "93b9082e-ac02-4490-a9aa-d805d7f9d737"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:16.953333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:16.953295 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93b9082e-ac02-4490-a9aa-d805d7f9d737-kube-api-access-2g6kt" pod "93b9082e-ac02-4490-a9aa-d805d7f9d737" (UID: "93b9082e-ac02-4490-a9aa-d805d7f9d737"). InnerVolumeSpecName "kube-api-access-2g6kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:16.955535 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:16.955502 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-metrics-certs" pod "93b9082e-ac02-4490-a9aa-d805d7f9d737" (UID: "93b9082e-ac02-4490-a9aa-d805d7f9d737"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:38:17.030617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.030576 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.030617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.030613 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93b9082e-ac02-4490-a9aa-d805d7f9d737-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.030617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.030625 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g6kt\" (UniqueName: \"kubernetes.io/projected/93b9082e-ac02-4490-a9aa-d805d7f9d737-kube-api-access-2g6kt\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.030880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.030635 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93b9082e-ac02-4490-a9aa-d805d7f9d737-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.195961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.195835 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:17.250616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.250576 4828 generic.go:356] "Generic (PLEG): container finished" podID="93b9082e-ac02-4490-a9aa-d805d7f9d737" containerID="b90658b778c40e407d8f54f499960045d86c839bd376f8e118a97a2d1f88adb1" exitCode=0 Oct 30 17:38:17.250802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.250646 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" Oct 30 17:38:17.250802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.250647 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" event={"ID":"93b9082e-ac02-4490-a9aa-d805d7f9d737","Type":"ContainerDied","Data":"b90658b778c40e407d8f54f499960045d86c839bd376f8e118a97a2d1f88adb1"} Oct 30 17:38:17.250802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.250716 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84" event={"ID":"93b9082e-ac02-4490-a9aa-d805d7f9d737","Type":"ContainerDied","Data":"9bcaa1b70d6426f90f295ada0359ba3cf9153dc9b82c3dd58ef234db764a8069"} Oct 30 17:38:17.250802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.250746 4828 scope.go:122] "RemoveContainer" containerID="b90658b778c40e407d8f54f499960045d86c839bd376f8e118a97a2d1f88adb1" Oct 30 17:38:17.261059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.261025 4828 scope.go:122] "RemoveContainer" containerID="b90658b778c40e407d8f54f499960045d86c839bd376f8e118a97a2d1f88adb1" Oct 30 17:38:17.261394 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:38:17.261369 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90658b778c40e407d8f54f499960045d86c839bd376f8e118a97a2d1f88adb1\": container with ID starting with b90658b778c40e407d8f54f499960045d86c839bd376f8e118a97a2d1f88adb1 not found: ID does not exist" containerID="b90658b778c40e407d8f54f499960045d86c839bd376f8e118a97a2d1f88adb1" Oct 30 17:38:17.261479 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.261402 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90658b778c40e407d8f54f499960045d86c839bd376f8e118a97a2d1f88adb1"} err="failed to get container status \"b90658b778c40e407d8f54f499960045d86c839bd376f8e118a97a2d1f88adb1\": rpc error: code = NotFound desc = could not find container \"b90658b778c40e407d8f54f499960045d86c839bd376f8e118a97a2d1f88adb1\": container with ID starting with b90658b778c40e407d8f54f499960045d86c839bd376f8e118a97a2d1f88adb1 not found: ID does not exist" Oct 30 17:38:17.309764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.309714 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84"] Oct 30 17:38:17.320185 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.320151 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-469ad7-6bb7cc94cb-pvv84"] Oct 30 17:38:17.349797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.349652 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:17.853637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:17.853606 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:18.491521 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:18.491487 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93b9082e-ac02-4490-a9aa-d805d7f9d737" path="/var/lib/kubelet/pods/93b9082e-ac02-4490-a9aa-d805d7f9d737/volumes" Oct 30 17:38:19.359070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:19.359024 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" podUID="f92a7745-1da0-468d-bbb0-fba201967378" containerName="payload-processing" containerID="cri-o://a8fbf4f4ab82eb5989f21c6e5feae7e9004924e09567bd07d3db734415b2b95f" gracePeriod=25 Oct 30 17:38:19.393320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:19.393271 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" podUID="1b3a84af-dd51-40b7-b426-b51d208ca058" containerName="payload-pre-processing" containerID="cri-o://12b5af56d271a5756c880e8d3d994946d7935d9814a0237f0689a39932105b93" gracePeriod=25 Oct 30 17:38:19.715913 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:19.715807 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:21.333264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:21.330130 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:22.942503 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:22.942461 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:24.403113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:24.403076 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:24.555539 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:24.555498 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:24.722124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:24.722026 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk"] Oct 30 17:38:24.722309 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:24.722240 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" podUID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" containerName="istio-proxy" containerID="cri-o://61eb605b9637101043a54e0e6301205b6315ad66004a216264f0f0143981a388" gracePeriod=30 Oct 30 17:38:24.732758 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:24.732719 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:24.882847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:24.882807 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:27.018960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:27.018876 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:27.463533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:27.463449 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" podUID="1b3a84af-dd51-40b7-b426-b51d208ca058" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:38:27.463533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:27.463511 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" podUID="f92a7745-1da0-468d-bbb0-fba201967378" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:38:28.757348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:28.757299 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:29.998357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:29.998292 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:38:30.100268 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.100226 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-kube-api-access-5qtpf\" (UniqueName: \"kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-kube-api-access-5qtpf\") pod \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " Oct 30 17:38:30.100471 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.100291 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-certs\") pod \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " Oct 30 17:38:30.100471 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.100349 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istiod-ca-cert\") pod \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " Oct 30 17:38:30.100604 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.100479 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-socket\") pod \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " Oct 30 17:38:30.100604 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.100542 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-token\" (UniqueName: \"kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-token\") pod \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " Oct 30 17:38:30.100701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.100631 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-certs" pod "3454fdaa-9f5d-40c0-9ce5-7a801a281024" (UID: "3454fdaa-9f5d-40c0-9ce5-7a801a281024"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.100701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.100643 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-envoy\") pod \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " Oct 30 17:38:30.100818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.100722 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-data\") pod \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " Oct 30 17:38:30.100818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.100755 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-podinfo\") pod \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " Oct 30 17:38:30.100818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.100785 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-credential-socket\") pod \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\" (UID: \"3454fdaa-9f5d-40c0-9ce5-7a801a281024\") " Oct 30 17:38:30.101265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.100779 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istiod-ca-cert" pod "3454fdaa-9f5d-40c0-9ce5-7a801a281024" (UID: "3454fdaa-9f5d-40c0-9ce5-7a801a281024"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:30.101265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.101069 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-data" pod "3454fdaa-9f5d-40c0-9ce5-7a801a281024" (UID: "3454fdaa-9f5d-40c0-9ce5-7a801a281024"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.101265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.101249 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-credential-socket" pod "3454fdaa-9f5d-40c0-9ce5-7a801a281024" (UID: "3454fdaa-9f5d-40c0-9ce5-7a801a281024"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.101515 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.101318 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.101515 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.101486 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-socket" pod "3454fdaa-9f5d-40c0-9ce5-7a801a281024" (UID: "3454fdaa-9f5d-40c0-9ce5-7a801a281024"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.102936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.102885 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-kube-api-access-5qtpf" pod "3454fdaa-9f5d-40c0-9ce5-7a801a281024" (UID: "3454fdaa-9f5d-40c0-9ce5-7a801a281024"). InnerVolumeSpecName "kube-api-access-5qtpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:30.103185 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.103153 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-token" pod "3454fdaa-9f5d-40c0-9ce5-7a801a281024" (UID: "3454fdaa-9f5d-40c0-9ce5-7a801a281024"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:30.103710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.103680 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-envoy" pod "3454fdaa-9f5d-40c0-9ce5-7a801a281024" (UID: "3454fdaa-9f5d-40c0-9ce5-7a801a281024"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:30.103852 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.103833 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-podinfo" pod "3454fdaa-9f5d-40c0-9ce5-7a801a281024" (UID: "3454fdaa-9f5d-40c0-9ce5-7a801a281024"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:30.202468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.202428 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.202468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.202460 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.202468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.202469 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.202719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.202480 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.202719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.202491 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.202719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.202503 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3454fdaa-9f5d-40c0-9ce5-7a801a281024-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.202719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.202513 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qtpf\" (UniqueName: \"kubernetes.io/projected/3454fdaa-9f5d-40c0-9ce5-7a801a281024-kube-api-access-5qtpf\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.202719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.202522 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3454fdaa-9f5d-40c0-9ce5-7a801a281024-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:30.343760 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.343721 4828 generic.go:356] "Generic (PLEG): container finished" podID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" containerID="61eb605b9637101043a54e0e6301205b6315ad66004a216264f0f0143981a388" exitCode=0 Oct 30 17:38:30.343968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.343775 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" Oct 30 17:38:30.343968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.343815 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" event={"ID":"3454fdaa-9f5d-40c0-9ce5-7a801a281024","Type":"ContainerDied","Data":"61eb605b9637101043a54e0e6301205b6315ad66004a216264f0f0143981a388"} Oct 30 17:38:30.343968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.343927 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk" event={"ID":"3454fdaa-9f5d-40c0-9ce5-7a801a281024","Type":"ContainerDied","Data":"22327d1fdc380c550021999db5f22355049491fb2be61fd3ac3faec44bf3a40a"} Oct 30 17:38:30.343968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.343964 4828 scope.go:122] "RemoveContainer" containerID="61eb605b9637101043a54e0e6301205b6315ad66004a216264f0f0143981a388" Oct 30 17:38:30.356711 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.356686 4828 scope.go:122] "RemoveContainer" containerID="61eb605b9637101043a54e0e6301205b6315ad66004a216264f0f0143981a388" Oct 30 17:38:30.357069 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:38:30.357048 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61eb605b9637101043a54e0e6301205b6315ad66004a216264f0f0143981a388\": container with ID starting with 61eb605b9637101043a54e0e6301205b6315ad66004a216264f0f0143981a388 not found: ID does not exist" containerID="61eb605b9637101043a54e0e6301205b6315ad66004a216264f0f0143981a388" Oct 30 17:38:30.357125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.357081 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61eb605b9637101043a54e0e6301205b6315ad66004a216264f0f0143981a388"} err="failed to get container status \"61eb605b9637101043a54e0e6301205b6315ad66004a216264f0f0143981a388\": rpc error: code = NotFound desc = could not find container \"61eb605b9637101043a54e0e6301205b6315ad66004a216264f0f0143981a388\": container with ID starting with 61eb605b9637101043a54e0e6301205b6315ad66004a216264f0f0143981a388 not found: ID does not exist" Oct 30 17:38:30.400800 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.400759 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk"] Oct 30 17:38:30.414235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.414163 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-469ad7-openshift-default-b4fd55565-54ftk"] Oct 30 17:38:30.447975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.447940 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:30.491622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:30.491589 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" path="/var/lib/kubelet/pods/3454fdaa-9f5d-40c0-9ce5-7a801a281024/volumes" Oct 30 17:38:32.191498 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:32.191410 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:34.249714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:34.247952 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:35.397690 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:35.397653 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn"] Oct 30 17:38:35.588009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:35.587968 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:35.757417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:35.757335 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:35.997850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:35.997811 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:37.124594 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.124551 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f"] Oct 30 17:38:37.125118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.124810 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" podUID="9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58" containerName="maas-api" containerID="cri-o://ab525f6463e826f5b955b858a017ffff48b57bc1f5cef15413567e2c6bcb3c33" gracePeriod=30 Oct 30 17:38:37.226202 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.226166 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:37.403064 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.402911 4828 generic.go:356] "Generic (PLEG): container finished" podID="9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58" containerID="ab525f6463e826f5b955b858a017ffff48b57bc1f5cef15413567e2c6bcb3c33" exitCode=0 Oct 30 17:38:37.403356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.403278 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" event={"ID":"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58","Type":"ContainerDied","Data":"ab525f6463e826f5b955b858a017ffff48b57bc1f5cef15413567e2c6bcb3c33"} Oct 30 17:38:37.432884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.432853 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:38:37.463144 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.463110 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" podUID="f92a7745-1da0-468d-bbb0-fba201967378" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:38:37.463289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.463217 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" podUID="1b3a84af-dd51-40b7-b426-b51d208ca058" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:38:37.498973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.496915 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-openshift-service-ca\") pod \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " Oct 30 17:38:37.498973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.497084 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-kube-api-access-kxh2x\" (UniqueName: \"kubernetes.io/projected/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-kube-api-access-kxh2x\") pod \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " Oct 30 17:38:37.498973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.497126 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-maas-api-tls\") pod \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " Oct 30 17:38:37.498973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.497183 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-metrics-certs\") pod \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\" (UID: \"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58\") " Oct 30 17:38:37.499990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.499638 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-openshift-service-ca" pod "9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58" (UID: "9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:37.519180 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.510273 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-maas-api-tls" pod "9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58" (UID: "9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:38:37.519180 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.519131 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-kube-api-access-kxh2x" pod "9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58" (UID: "9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58"). InnerVolumeSpecName "kube-api-access-kxh2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:37.519782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.519753 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-metrics-certs" pod "9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58" (UID: "9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:38:37.598963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.598894 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxh2x\" (UniqueName: \"kubernetes.io/projected/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-kube-api-access-kxh2x\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:37.598963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.598963 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:37.598963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.598973 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:37.599210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.598982 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:37.824363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.824325 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:37.912226 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:38:37.912191 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:38:37.992842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:37.992803 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:38.146804 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:38.146713 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:38.411364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:38.411280 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" Oct 30 17:38:38.411364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:38.411315 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f" event={"ID":"9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58","Type":"ContainerDied","Data":"76c2074c6598416ced2e67d3ebe3c35faca8c5a0ce39c16095d9d9fe2eb3ccd0"} Oct 30 17:38:38.411569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:38.411380 4828 scope.go:122] "RemoveContainer" containerID="ab525f6463e826f5b955b858a017ffff48b57bc1f5cef15413567e2c6bcb3c33" Oct 30 17:38:38.465159 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:38.465124 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f"] Oct 30 17:38:38.477507 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:38.477462 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-75586d-569454d859-7gw2f"] Oct 30 17:38:38.488425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:38.488393 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:38.490611 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:38.490582 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58" path="/var/lib/kubelet/pods/9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58/volumes" Oct 30 17:38:39.684613 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.684582 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mj8j7_c055e279-12c7-4707-b602-56548f5df870/payload-processing/0.log" Oct 30 17:38:39.685367 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.685345 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" Oct 30 17:38:39.709794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.709767 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" Oct 30 17:38:39.718023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.717993 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f92a7745-1da0-468d-bbb0-fba201967378-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92a7745-1da0-468d-bbb0-fba201967378-plugins-config-volume\") pod \"f92a7745-1da0-468d-bbb0-fba201967378\" (UID: \"f92a7745-1da0-468d-bbb0-fba201967378\") " Oct 30 17:38:39.718218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.718047 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3a84af-dd51-40b7-b426-b51d208ca058-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3a84af-dd51-40b7-b426-b51d208ca058-plugins-config-volume\") pod \"1b3a84af-dd51-40b7-b426-b51d208ca058\" (UID: \"1b3a84af-dd51-40b7-b426-b51d208ca058\") " Oct 30 17:38:39.718218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.718105 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f92a7745-1da0-468d-bbb0-fba201967378-kube-api-access-kpjmq\" (UniqueName: \"kubernetes.io/projected/f92a7745-1da0-468d-bbb0-fba201967378-kube-api-access-kpjmq\") pod \"f92a7745-1da0-468d-bbb0-fba201967378\" (UID: \"f92a7745-1da0-468d-bbb0-fba201967378\") " Oct 30 17:38:39.718218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.718152 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3a84af-dd51-40b7-b426-b51d208ca058-kube-api-access-n6cfq\" (UniqueName: \"kubernetes.io/projected/1b3a84af-dd51-40b7-b426-b51d208ca058-kube-api-access-n6cfq\") pod \"1b3a84af-dd51-40b7-b426-b51d208ca058\" (UID: \"1b3a84af-dd51-40b7-b426-b51d208ca058\") " Oct 30 17:38:39.718453 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.718412 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f92a7745-1da0-468d-bbb0-fba201967378-plugins-config-volume" pod "f92a7745-1da0-468d-bbb0-fba201967378" (UID: "f92a7745-1da0-468d-bbb0-fba201967378"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:39.718536 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.718485 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3a84af-dd51-40b7-b426-b51d208ca058-plugins-config-volume" pod "1b3a84af-dd51-40b7-b426-b51d208ca058" (UID: "1b3a84af-dd51-40b7-b426-b51d208ca058"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:39.718643 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.718626 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92a7745-1da0-468d-bbb0-fba201967378-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:39.718708 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.718645 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3a84af-dd51-40b7-b426-b51d208ca058-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:39.720247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.720226 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3a84af-dd51-40b7-b426-b51d208ca058-kube-api-access-n6cfq" pod "1b3a84af-dd51-40b7-b426-b51d208ca058" (UID: "1b3a84af-dd51-40b7-b426-b51d208ca058"). InnerVolumeSpecName "kube-api-access-n6cfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:39.720381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.720346 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92a7745-1da0-468d-bbb0-fba201967378-kube-api-access-kpjmq" pod "f92a7745-1da0-468d-bbb0-fba201967378" (UID: "f92a7745-1da0-468d-bbb0-fba201967378"). InnerVolumeSpecName "kube-api-access-kpjmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:39.820213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.820176 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpjmq\" (UniqueName: \"kubernetes.io/projected/f92a7745-1da0-468d-bbb0-fba201967378-kube-api-access-kpjmq\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:39.820213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.820207 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6cfq\" (UniqueName: \"kubernetes.io/projected/1b3a84af-dd51-40b7-b426-b51d208ca058-kube-api-access-n6cfq\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:39.889069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:39.888985 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mj8j7_c055e279-12c7-4707-b602-56548f5df870/payload-processing/0.log" Oct 30 17:38:40.228025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.226484 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:40.398562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.398516 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" podUID="c64dc903-607a-4627-92a7-74f18599c40b" containerName="payload-processing" containerID="cri-o://10b13c58d4418eeac94bc456d8adf334803974f758c03bb0ff9e9c608bead54f" gracePeriod=25 Oct 30 17:38:40.428653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.428614 4828 generic.go:356] "Generic (PLEG): container finished" podID="1b3a84af-dd51-40b7-b426-b51d208ca058" containerID="12b5af56d271a5756c880e8d3d994946d7935d9814a0237f0689a39932105b93" exitCode=0 Oct 30 17:38:40.428815 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.428678 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" Oct 30 17:38:40.428815 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.428689 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" event={"ID":"1b3a84af-dd51-40b7-b426-b51d208ca058","Type":"ContainerDied","Data":"12b5af56d271a5756c880e8d3d994946d7935d9814a0237f0689a39932105b93"} Oct 30 17:38:40.428815 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.428775 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75" event={"ID":"1b3a84af-dd51-40b7-b426-b51d208ca058","Type":"ContainerDied","Data":"437e6c43b02d97c458281989d7c4cd2ac339106376fe438fceda3fb7496bb4cc"} Oct 30 17:38:40.428815 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.428804 4828 scope.go:122] "RemoveContainer" containerID="12b5af56d271a5756c880e8d3d994946d7935d9814a0237f0689a39932105b93" Oct 30 17:38:40.430148 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.430126 4828 generic.go:356] "Generic (PLEG): container finished" podID="f92a7745-1da0-468d-bbb0-fba201967378" containerID="a8fbf4f4ab82eb5989f21c6e5feae7e9004924e09567bd07d3db734415b2b95f" exitCode=0 Oct 30 17:38:40.430246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.430175 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" Oct 30 17:38:40.430246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.430174 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" event={"ID":"f92a7745-1da0-468d-bbb0-fba201967378","Type":"ContainerDied","Data":"a8fbf4f4ab82eb5989f21c6e5feae7e9004924e09567bd07d3db734415b2b95f"} Oct 30 17:38:40.430246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.430235 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7" event={"ID":"f92a7745-1da0-468d-bbb0-fba201967378","Type":"ContainerDied","Data":"469ba3e2f9621f97eded072ff5372692c7e47b7b39b10f758648ccca0f5f69c4"} Oct 30 17:38:40.441074 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.441053 4828 scope.go:122] "RemoveContainer" containerID="12b5af56d271a5756c880e8d3d994946d7935d9814a0237f0689a39932105b93" Oct 30 17:38:40.441400 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:38:40.441370 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b5af56d271a5756c880e8d3d994946d7935d9814a0237f0689a39932105b93\": container with ID starting with 12b5af56d271a5756c880e8d3d994946d7935d9814a0237f0689a39932105b93 not found: ID does not exist" containerID="12b5af56d271a5756c880e8d3d994946d7935d9814a0237f0689a39932105b93" Oct 30 17:38:40.441464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.441413 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b5af56d271a5756c880e8d3d994946d7935d9814a0237f0689a39932105b93"} err="failed to get container status \"12b5af56d271a5756c880e8d3d994946d7935d9814a0237f0689a39932105b93\": rpc error: code = NotFound desc = could not find container \"12b5af56d271a5756c880e8d3d994946d7935d9814a0237f0689a39932105b93\": container with ID starting with 12b5af56d271a5756c880e8d3d994946d7935d9814a0237f0689a39932105b93 not found: ID does not exist" Oct 30 17:38:40.441464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.441438 4828 scope.go:122] "RemoveContainer" containerID="a8fbf4f4ab82eb5989f21c6e5feae7e9004924e09567bd07d3db734415b2b95f" Oct 30 17:38:40.450715 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.450692 4828 scope.go:122] "RemoveContainer" containerID="a8fbf4f4ab82eb5989f21c6e5feae7e9004924e09567bd07d3db734415b2b95f" Oct 30 17:38:40.451033 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:38:40.451012 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8fbf4f4ab82eb5989f21c6e5feae7e9004924e09567bd07d3db734415b2b95f\": container with ID starting with a8fbf4f4ab82eb5989f21c6e5feae7e9004924e09567bd07d3db734415b2b95f not found: ID does not exist" containerID="a8fbf4f4ab82eb5989f21c6e5feae7e9004924e09567bd07d3db734415b2b95f" Oct 30 17:38:40.451091 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.451043 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8fbf4f4ab82eb5989f21c6e5feae7e9004924e09567bd07d3db734415b2b95f"} err="failed to get container status \"a8fbf4f4ab82eb5989f21c6e5feae7e9004924e09567bd07d3db734415b2b95f\": rpc error: code = NotFound desc = could not find container \"a8fbf4f4ab82eb5989f21c6e5feae7e9004924e09567bd07d3db734415b2b95f\": container with ID starting with a8fbf4f4ab82eb5989f21c6e5feae7e9004924e09567bd07d3db734415b2b95f not found: ID does not exist" Oct 30 17:38:40.490410 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.490333 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75"] Oct 30 17:38:40.497759 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.497723 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-469ad7-5696dbbb8c-p7j75"] Oct 30 17:38:40.532323 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.532287 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7"] Oct 30 17:38:40.540740 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:40.540697 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-469ad7-5954b799d-w4wr7"] Oct 30 17:38:42.082878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:42.081322 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:42.490272 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:42.490196 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3a84af-dd51-40b7-b426-b51d208ca058" path="/var/lib/kubelet/pods/1b3a84af-dd51-40b7-b426-b51d208ca058/volumes" Oct 30 17:38:42.490604 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:42.490591 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f92a7745-1da0-468d-bbb0-fba201967378" path="/var/lib/kubelet/pods/f92a7745-1da0-468d-bbb0-fba201967378/volumes" Oct 30 17:38:43.859495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:43.856798 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:44.372416 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:44.372378 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" podUID="c64dc903-607a-4627-92a7-74f18599c40b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:38:45.323243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:45.323204 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:45.499347 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:45.499300 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:45.617262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:45.617174 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt"] Oct 30 17:38:45.617510 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:45.617456 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" podUID="5eb25a1e-9f6b-4823-982a-82b25909fd53" containerName="istio-proxy" containerID="cri-o://060d6f1dd72bcd8266816e29075b16fb647825c7f84e68563a944b027dabee91" gracePeriod=30 Oct 30 17:38:45.656951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:45.656875 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:45.826758 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:45.826718 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:46.161164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:46.161128 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:47.908403 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:47.908364 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:49.662802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:49.662729 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:50.015989 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.015951 4828 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:38:50.016187 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.016049 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" podUID="5eb25a1e-9f6b-4823-982a-82b25909fd53" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:38:50.883675 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.883591 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:38:50.954465 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.954422 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-kube-api-access-6phkn\" (UniqueName: \"kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-kube-api-access-6phkn\") pod \"5eb25a1e-9f6b-4823-982a-82b25909fd53\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " Oct 30 17:38:50.954776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.954529 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-credential-socket\") pod \"5eb25a1e-9f6b-4823-982a-82b25909fd53\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " Oct 30 17:38:50.954776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.954566 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-socket\") pod \"5eb25a1e-9f6b-4823-982a-82b25909fd53\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " Oct 30 17:38:50.954776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.954671 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-token\" (UniqueName: \"kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-token\") pod \"5eb25a1e-9f6b-4823-982a-82b25909fd53\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " Oct 30 17:38:50.954776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.954711 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-data\") pod \"5eb25a1e-9f6b-4823-982a-82b25909fd53\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " Oct 30 17:38:50.954776 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.954751 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-certs\") pod \"5eb25a1e-9f6b-4823-982a-82b25909fd53\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " Oct 30 17:38:50.955164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.954785 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-envoy\") pod \"5eb25a1e-9f6b-4823-982a-82b25909fd53\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " Oct 30 17:38:50.955164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.954825 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb25a1e-9f6b-4823-982a-82b25909fd53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb25a1e-9f6b-4823-982a-82b25909fd53-istiod-ca-cert\") pod \"5eb25a1e-9f6b-4823-982a-82b25909fd53\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " Oct 30 17:38:50.955164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.954887 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-podinfo\") pod \"5eb25a1e-9f6b-4823-982a-82b25909fd53\" (UID: \"5eb25a1e-9f6b-4823-982a-82b25909fd53\") " Oct 30 17:38:50.955164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.954953 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-socket" pod "5eb25a1e-9f6b-4823-982a-82b25909fd53" (UID: "5eb25a1e-9f6b-4823-982a-82b25909fd53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:50.955164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.955134 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-data" pod "5eb25a1e-9f6b-4823-982a-82b25909fd53" (UID: "5eb25a1e-9f6b-4823-982a-82b25909fd53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:50.955438 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.955422 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:50.955500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.955442 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:50.955840 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.955813 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-credential-socket" pod "5eb25a1e-9f6b-4823-982a-82b25909fd53" (UID: "5eb25a1e-9f6b-4823-982a-82b25909fd53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:50.956264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.956238 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb25a1e-9f6b-4823-982a-82b25909fd53-istiod-ca-cert" pod "5eb25a1e-9f6b-4823-982a-82b25909fd53" (UID: "5eb25a1e-9f6b-4823-982a-82b25909fd53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:50.956521 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.956496 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-certs" pod "5eb25a1e-9f6b-4823-982a-82b25909fd53" (UID: "5eb25a1e-9f6b-4823-982a-82b25909fd53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:50.957858 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.957832 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-podinfo" pod "5eb25a1e-9f6b-4823-982a-82b25909fd53" (UID: "5eb25a1e-9f6b-4823-982a-82b25909fd53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:50.958346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.958317 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-kube-api-access-6phkn" pod "5eb25a1e-9f6b-4823-982a-82b25909fd53" (UID: "5eb25a1e-9f6b-4823-982a-82b25909fd53"). InnerVolumeSpecName "kube-api-access-6phkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:50.958453 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.958424 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-token" pod "5eb25a1e-9f6b-4823-982a-82b25909fd53" (UID: "5eb25a1e-9f6b-4823-982a-82b25909fd53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:50.959237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:50.959193 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-envoy" pod "5eb25a1e-9f6b-4823-982a-82b25909fd53" (UID: "5eb25a1e-9f6b-4823-982a-82b25909fd53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:51.056853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.056819 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6phkn\" (UniqueName: \"kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-kube-api-access-6phkn\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:51.056853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.056850 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:51.056853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.056861 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:51.057104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.056871 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:51.057104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.056881 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:51.057104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.056889 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb25a1e-9f6b-4823-982a-82b25909fd53-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:51.057104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.056908 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb25a1e-9f6b-4823-982a-82b25909fd53-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:38:51.085398 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.085362 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw"] Oct 30 17:38:51.258176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.258144 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:51.416802 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.416766 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:51.502110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.502075 4828 generic.go:356] "Generic (PLEG): container finished" podID="5eb25a1e-9f6b-4823-982a-82b25909fd53" containerID="060d6f1dd72bcd8266816e29075b16fb647825c7f84e68563a944b027dabee91" exitCode=0 Oct 30 17:38:51.502299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.502140 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" Oct 30 17:38:51.502299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.502159 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" event={"ID":"5eb25a1e-9f6b-4823-982a-82b25909fd53","Type":"ContainerDied","Data":"060d6f1dd72bcd8266816e29075b16fb647825c7f84e68563a944b027dabee91"} Oct 30 17:38:51.502299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.502231 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt" event={"ID":"5eb25a1e-9f6b-4823-982a-82b25909fd53","Type":"ContainerDied","Data":"b3db9eb231aff2f892b036776e65c5d19c5790cc3d4e52b0e98c0b0ba9d4261c"} Oct 30 17:38:51.502299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.502260 4828 scope.go:122] "RemoveContainer" containerID="060d6f1dd72bcd8266816e29075b16fb647825c7f84e68563a944b027dabee91" Oct 30 17:38:51.514304 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.514276 4828 scope.go:122] "RemoveContainer" containerID="060d6f1dd72bcd8266816e29075b16fb647825c7f84e68563a944b027dabee91" Oct 30 17:38:51.514637 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:38:51.514614 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060d6f1dd72bcd8266816e29075b16fb647825c7f84e68563a944b027dabee91\": container with ID starting with 060d6f1dd72bcd8266816e29075b16fb647825c7f84e68563a944b027dabee91 not found: ID does not exist" containerID="060d6f1dd72bcd8266816e29075b16fb647825c7f84e68563a944b027dabee91" Oct 30 17:38:51.514747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.514647 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060d6f1dd72bcd8266816e29075b16fb647825c7f84e68563a944b027dabee91"} err="failed to get container status \"060d6f1dd72bcd8266816e29075b16fb647825c7f84e68563a944b027dabee91\": rpc error: code = NotFound desc = could not find container \"060d6f1dd72bcd8266816e29075b16fb647825c7f84e68563a944b027dabee91\": container with ID starting with 060d6f1dd72bcd8266816e29075b16fb647825c7f84e68563a944b027dabee91 not found: ID does not exist" Oct 30 17:38:51.570171 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.570137 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt"] Oct 30 17:38:51.583338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:51.583304 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75586d-openshift-default-5858999787-7ksbt"] Oct 30 17:38:52.489448 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:52.489417 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb25a1e-9f6b-4823-982a-82b25909fd53" path="/var/lib/kubelet/pods/5eb25a1e-9f6b-4823-982a-82b25909fd53/volumes" Oct 30 17:38:53.354046 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:53.353797 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:54.373764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:54.373712 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" podUID="c64dc903-607a-4627-92a7-74f18599c40b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:38:54.896523 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:54.894986 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:56.444731 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:56.444680 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45"] Oct 30 17:38:56.499283 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:56.499245 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:56.670156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:56.670116 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:38:58.720879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:38:58.720837 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:00.538845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:00.538811 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:00.579972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:00.579943 4828 generic.go:356] "Generic (PLEG): container finished" podID="c64dc903-607a-4627-92a7-74f18599c40b" containerID="10b13c58d4418eeac94bc456d8adf334803974f758c03bb0ff9e9c608bead54f" exitCode=0 Oct 30 17:39:00.580212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:00.580092 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" event={"ID":"c64dc903-607a-4627-92a7-74f18599c40b","Type":"ContainerDied","Data":"10b13c58d4418eeac94bc456d8adf334803974f758c03bb0ff9e9c608bead54f"} Oct 30 17:39:00.682669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:00.682637 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" Oct 30 17:39:00.695915 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:00.695876 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:00.770271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:00.770237 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64dc903-607a-4627-92a7-74f18599c40b-kube-api-access-czs2t\" (UniqueName: \"kubernetes.io/projected/c64dc903-607a-4627-92a7-74f18599c40b-kube-api-access-czs2t\") pod \"c64dc903-607a-4627-92a7-74f18599c40b\" (UID: \"c64dc903-607a-4627-92a7-74f18599c40b\") " Oct 30 17:39:00.770584 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:00.770311 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64dc903-607a-4627-92a7-74f18599c40b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64dc903-607a-4627-92a7-74f18599c40b-plugins-config-volume\") pod \"c64dc903-607a-4627-92a7-74f18599c40b\" (UID: \"c64dc903-607a-4627-92a7-74f18599c40b\") " Oct 30 17:39:00.770858 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:00.770834 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64dc903-607a-4627-92a7-74f18599c40b-plugins-config-volume" pod "c64dc903-607a-4627-92a7-74f18599c40b" (UID: "c64dc903-607a-4627-92a7-74f18599c40b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:00.773075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:00.773046 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64dc903-607a-4627-92a7-74f18599c40b-kube-api-access-czs2t" pod "c64dc903-607a-4627-92a7-74f18599c40b" (UID: "c64dc903-607a-4627-92a7-74f18599c40b"). InnerVolumeSpecName "kube-api-access-czs2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:00.872042 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:00.872008 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czs2t\" (UniqueName: \"kubernetes.io/projected/c64dc903-607a-4627-92a7-74f18599c40b-kube-api-access-czs2t\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:00.872042 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:00.872038 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64dc903-607a-4627-92a7-74f18599c40b-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:01.183584 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:01.183543 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:01.445522 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:01.445426 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" podUID="cf528e56-af27-46d6-8f3d-20cc44101152" containerName="payload-pre-processing" containerID="cri-o://d4bb4bd6bc1d280252c085b76de47cb11c2c372c86dee8e90827733b949c97d4" gracePeriod=25 Oct 30 17:39:01.587738 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:01.587702 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" Oct 30 17:39:01.588224 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:01.587711 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn" event={"ID":"c64dc903-607a-4627-92a7-74f18599c40b","Type":"ContainerDied","Data":"8ddf1333bb44a74cd7caede278763a68edc91c97cec84ee76c66471014fe9a86"} Oct 30 17:39:01.588224 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:01.587840 4828 scope.go:122] "RemoveContainer" containerID="10b13c58d4418eeac94bc456d8adf334803974f758c03bb0ff9e9c608bead54f" Oct 30 17:39:01.658058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:01.658012 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn"] Oct 30 17:39:01.661441 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:01.661409 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75586d-645bb98c6-ckgcn"] Oct 30 17:39:02.220936 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:02.220881 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" podUID="cf528e56-af27-46d6-8f3d-20cc44101152" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:39:02.366293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:02.366230 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:02.491324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:02.491248 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64dc903-607a-4627-92a7-74f18599c40b" path="/var/lib/kubelet/pods/c64dc903-607a-4627-92a7-74f18599c40b/volumes" Oct 30 17:39:04.108921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:04.108695 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:05.420058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:05.419620 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr"] Oct 30 17:39:05.420058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:05.419967 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" podUID="d0ee8b49-7ba2-485a-8f44-92f365ea306f" containerName="maas-api" containerID="cri-o://62f5592054104d51e45c28c73215a0bd58380d27e55e334d7573fac57789907a" gracePeriod=30 Oct 30 17:39:05.532739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:05.532687 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:05.718334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:05.718268 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:05.900247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:05.900124 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:05.940138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:05.940104 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:39:06.106733 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.106687 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" containerID="cri-o://ab32c091c022325c2c6d3f1b974722dcaafb9e6f4deff6eba89e0b4bbe2d52d1" gracePeriod=45 Oct 30 17:39:06.128823 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.128772 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-maas-api-tls\") pod \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " Oct 30 17:39:06.128974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.128913 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ee8b49-7ba2-485a-8f44-92f365ea306f-kube-api-access-zfcdp\" (UniqueName: \"kubernetes.io/projected/d0ee8b49-7ba2-485a-8f44-92f365ea306f-kube-api-access-zfcdp\") pod \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " Oct 30 17:39:06.128974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.128961 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ee8b49-7ba2-485a-8f44-92f365ea306f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee8b49-7ba2-485a-8f44-92f365ea306f-openshift-service-ca\") pod \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " Oct 30 17:39:06.129125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.129023 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-metrics-certs\") pod \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\" (UID: \"d0ee8b49-7ba2-485a-8f44-92f365ea306f\") " Oct 30 17:39:06.129644 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.129614 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ee8b49-7ba2-485a-8f44-92f365ea306f-openshift-service-ca" pod "d0ee8b49-7ba2-485a-8f44-92f365ea306f" (UID: "d0ee8b49-7ba2-485a-8f44-92f365ea306f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:06.131249 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.131222 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-maas-api-tls" pod "d0ee8b49-7ba2-485a-8f44-92f365ea306f" (UID: "d0ee8b49-7ba2-485a-8f44-92f365ea306f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:39:06.131379 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.131360 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ee8b49-7ba2-485a-8f44-92f365ea306f-kube-api-access-zfcdp" pod "d0ee8b49-7ba2-485a-8f44-92f365ea306f" (UID: "d0ee8b49-7ba2-485a-8f44-92f365ea306f"). InnerVolumeSpecName "kube-api-access-zfcdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:06.131468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.131454 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-metrics-certs" pod "d0ee8b49-7ba2-485a-8f44-92f365ea306f" (UID: "d0ee8b49-7ba2-485a-8f44-92f365ea306f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:39:06.189861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.189826 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:06.230163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.230138 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.230163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.230162 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfcdp\" (UniqueName: \"kubernetes.io/projected/d0ee8b49-7ba2-485a-8f44-92f365ea306f-kube-api-access-zfcdp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.230329 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.230176 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ee8b49-7ba2-485a-8f44-92f365ea306f-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.230329 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.230187 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0ee8b49-7ba2-485a-8f44-92f365ea306f-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.341173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.341136 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:06.369866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.369835 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:39:06.491923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.491404 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:06.534465 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.534426 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-tmp-dir\") pod \"31569569-b53b-43c8-a6ca-2970efc5e331\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " Oct 30 17:39:06.534932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.534477 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31569569-b53b-43c8-a6ca-2970efc5e331-kube-api-access-6bpv4\" (UniqueName: \"kubernetes.io/projected/31569569-b53b-43c8-a6ca-2970efc5e331-kube-api-access-6bpv4\") pod \"31569569-b53b-43c8-a6ca-2970efc5e331\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " Oct 30 17:39:06.534932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.534522 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-home\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-home\") pod \"31569569-b53b-43c8-a6ca-2970efc5e331\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " Oct 30 17:39:06.534932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.534592 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-model-cache\") pod \"31569569-b53b-43c8-a6ca-2970efc5e331\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " Oct 30 17:39:06.534932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.534710 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31569569-b53b-43c8-a6ca-2970efc5e331-tls-certs\" (UniqueName: \"kubernetes.io/secret/31569569-b53b-43c8-a6ca-2970efc5e331-tls-certs\") pod \"31569569-b53b-43c8-a6ca-2970efc5e331\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " Oct 30 17:39:06.534932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.534739 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-dshm\") pod \"31569569-b53b-43c8-a6ca-2970efc5e331\" (UID: \"31569569-b53b-43c8-a6ca-2970efc5e331\") " Oct 30 17:39:06.534932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.534785 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-tmp-dir" pod "31569569-b53b-43c8-a6ca-2970efc5e331" (UID: "31569569-b53b-43c8-a6ca-2970efc5e331"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:06.535382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.535011 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-model-cache" pod "31569569-b53b-43c8-a6ca-2970efc5e331" (UID: "31569569-b53b-43c8-a6ca-2970efc5e331"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:06.535382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.535123 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-home" pod "31569569-b53b-43c8-a6ca-2970efc5e331" (UID: "31569569-b53b-43c8-a6ca-2970efc5e331"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:06.535382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.535242 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.535382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.535263 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.535382 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.535277 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.538005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.537981 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-dshm" pod "31569569-b53b-43c8-a6ca-2970efc5e331" (UID: "31569569-b53b-43c8-a6ca-2970efc5e331"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:06.538147 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.538016 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31569569-b53b-43c8-a6ca-2970efc5e331-kube-api-access-6bpv4" pod "31569569-b53b-43c8-a6ca-2970efc5e331" (UID: "31569569-b53b-43c8-a6ca-2970efc5e331"). InnerVolumeSpecName "kube-api-access-6bpv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:06.538364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.538350 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31569569-b53b-43c8-a6ca-2970efc5e331-tls-certs" pod "31569569-b53b-43c8-a6ca-2970efc5e331" (UID: "31569569-b53b-43c8-a6ca-2970efc5e331"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:39:06.626025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.625942 4828 generic.go:356] "Generic (PLEG): container finished" podID="d0ee8b49-7ba2-485a-8f44-92f365ea306f" containerID="62f5592054104d51e45c28c73215a0bd58380d27e55e334d7573fac57789907a" exitCode=0 Oct 30 17:39:06.626025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.625978 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" Oct 30 17:39:06.626025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.626014 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" event={"ID":"d0ee8b49-7ba2-485a-8f44-92f365ea306f","Type":"ContainerDied","Data":"62f5592054104d51e45c28c73215a0bd58380d27e55e334d7573fac57789907a"} Oct 30 17:39:06.626318 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.626070 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr" event={"ID":"d0ee8b49-7ba2-485a-8f44-92f365ea306f","Type":"ContainerDied","Data":"7d1db1393ab65641c07a0fdbddee8da84d4b1e488c82822eb710f5f447ba810b"} Oct 30 17:39:06.626318 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.626087 4828 scope.go:122] "RemoveContainer" containerID="62f5592054104d51e45c28c73215a0bd58380d27e55e334d7573fac57789907a" Oct 30 17:39:06.627810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.627781 4828 generic.go:356] "Generic (PLEG): container finished" podID="31569569-b53b-43c8-a6ca-2970efc5e331" containerID="ab32c091c022325c2c6d3f1b974722dcaafb9e6f4deff6eba89e0b4bbe2d52d1" exitCode=0 Oct 30 17:39:06.627961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.627847 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" Oct 30 17:39:06.627961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.627891 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" event={"ID":"31569569-b53b-43c8-a6ca-2970efc5e331","Type":"ContainerDied","Data":"ab32c091c022325c2c6d3f1b974722dcaafb9e6f4deff6eba89e0b4bbe2d52d1"} Oct 30 17:39:06.628103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.627963 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw" event={"ID":"31569569-b53b-43c8-a6ca-2970efc5e331","Type":"ContainerDied","Data":"3c372a8af3e5e59fdb452cca7bf64594087c7d0e2819adc7d762c701ef599c93"} Oct 30 17:39:06.637076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.637030 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31569569-b53b-43c8-a6ca-2970efc5e331-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.637076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.637055 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31569569-b53b-43c8-a6ca-2970efc5e331-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.637076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.637070 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bpv4\" (UniqueName: \"kubernetes.io/projected/31569569-b53b-43c8-a6ca-2970efc5e331-kube-api-access-6bpv4\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:06.642516 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.642485 4828 scope.go:122] "RemoveContainer" containerID="62f5592054104d51e45c28c73215a0bd58380d27e55e334d7573fac57789907a" Oct 30 17:39:06.643034 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:39:06.642995 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f5592054104d51e45c28c73215a0bd58380d27e55e334d7573fac57789907a\": container with ID starting with 62f5592054104d51e45c28c73215a0bd58380d27e55e334d7573fac57789907a not found: ID does not exist" containerID="62f5592054104d51e45c28c73215a0bd58380d27e55e334d7573fac57789907a" Oct 30 17:39:06.643157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.643042 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f5592054104d51e45c28c73215a0bd58380d27e55e334d7573fac57789907a"} err="failed to get container status \"62f5592054104d51e45c28c73215a0bd58380d27e55e334d7573fac57789907a\": rpc error: code = NotFound desc = could not find container \"62f5592054104d51e45c28c73215a0bd58380d27e55e334d7573fac57789907a\": container with ID starting with 62f5592054104d51e45c28c73215a0bd58380d27e55e334d7573fac57789907a not found: ID does not exist" Oct 30 17:39:06.643157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.643067 4828 scope.go:122] "RemoveContainer" containerID="ab32c091c022325c2c6d3f1b974722dcaafb9e6f4deff6eba89e0b4bbe2d52d1" Oct 30 17:39:06.653997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.653975 4828 scope.go:122] "RemoveContainer" containerID="ab32c091c022325c2c6d3f1b974722dcaafb9e6f4deff6eba89e0b4bbe2d52d1" Oct 30 17:39:06.654299 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:39:06.654281 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab32c091c022325c2c6d3f1b974722dcaafb9e6f4deff6eba89e0b4bbe2d52d1\": container with ID starting with ab32c091c022325c2c6d3f1b974722dcaafb9e6f4deff6eba89e0b4bbe2d52d1 not found: ID does not exist" containerID="ab32c091c022325c2c6d3f1b974722dcaafb9e6f4deff6eba89e0b4bbe2d52d1" Oct 30 17:39:06.654377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.654307 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab32c091c022325c2c6d3f1b974722dcaafb9e6f4deff6eba89e0b4bbe2d52d1"} err="failed to get container status \"ab32c091c022325c2c6d3f1b974722dcaafb9e6f4deff6eba89e0b4bbe2d52d1\": rpc error: code = NotFound desc = could not find container \"ab32c091c022325c2c6d3f1b974722dcaafb9e6f4deff6eba89e0b4bbe2d52d1\": container with ID starting with ab32c091c022325c2c6d3f1b974722dcaafb9e6f4deff6eba89e0b4bbe2d52d1 not found: ID does not exist" Oct 30 17:39:06.681823 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.681786 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr"] Oct 30 17:39:06.689885 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.689854 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8f92ef-56588d7fdc-zn9zr"] Oct 30 17:39:06.721837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.721802 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw"] Oct 30 17:39:06.724422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.724403 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-096d75/distinct-2-w0-096d75-kserve-6c7d886775-5m9cw"] Oct 30 17:39:06.812274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:06.812242 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:07.912610 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:39:07.912572 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:39:08.490243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:08.490207 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" path="/var/lib/kubelet/pods/31569569-b53b-43c8-a6ca-2970efc5e331/volumes" Oct 30 17:39:08.490651 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:08.490637 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ee8b49-7ba2-485a-8f44-92f365ea306f" path="/var/lib/kubelet/pods/d0ee8b49-7ba2-485a-8f44-92f365ea306f/volumes" Oct 30 17:39:08.588586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:08.588550 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:10.225612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:10.222655 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:11.845305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:11.845271 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:12.157835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:12.157688 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:12.221545 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:12.221505 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" podUID="cf528e56-af27-46d6-8f3d-20cc44101152" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:39:13.704621 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:13.704582 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:14.919090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:14.919050 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp"] Oct 30 17:39:14.919714 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:14.919384 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" podUID="e5f0f53d-5f1a-4068-bea3-185bb56ab9ff" containerName="maas-api" containerID="cri-o://85ae1bb8a04c850d5d0f8794619b2ba957c4026d5cdebed9724e5be91aae263f" gracePeriod=30 Oct 30 17:39:15.012377 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.012337 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:15.214863 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.214828 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 17:39:15.230060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.230023 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-openshift-service-ca\") pod \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " Oct 30 17:39:15.230265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.230195 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-maas-api-tls\") pod \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " Oct 30 17:39:15.230265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.230261 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-metrics-certs\") pod \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " Oct 30 17:39:15.230387 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.230344 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-kube-api-access-qtchj\" (UniqueName: \"kubernetes.io/projected/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-kube-api-access-qtchj\") pod \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\" (UID: \"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff\") " Oct 30 17:39:15.230456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.230405 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-openshift-service-ca" pod "e5f0f53d-5f1a-4068-bea3-185bb56ab9ff" (UID: "e5f0f53d-5f1a-4068-bea3-185bb56ab9ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:15.230710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.230686 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:15.233759 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.233721 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-kube-api-access-qtchj" pod "e5f0f53d-5f1a-4068-bea3-185bb56ab9ff" (UID: "e5f0f53d-5f1a-4068-bea3-185bb56ab9ff"). InnerVolumeSpecName "kube-api-access-qtchj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:15.233919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.233718 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-maas-api-tls" pod "e5f0f53d-5f1a-4068-bea3-185bb56ab9ff" (UID: "e5f0f53d-5f1a-4068-bea3-185bb56ab9ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:39:15.233919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.233755 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-metrics-certs" pod "e5f0f53d-5f1a-4068-bea3-185bb56ab9ff" (UID: "e5f0f53d-5f1a-4068-bea3-185bb56ab9ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:39:15.338515 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.331803 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:15.338515 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.331844 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtchj\" (UniqueName: \"kubernetes.io/projected/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-kube-api-access-qtchj\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:15.338515 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.331861 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:15.516597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.516559 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:15.710932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.710876 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:15.719303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.719266 4828 generic.go:356] "Generic (PLEG): container finished" podID="e5f0f53d-5f1a-4068-bea3-185bb56ab9ff" containerID="85ae1bb8a04c850d5d0f8794619b2ba957c4026d5cdebed9724e5be91aae263f" exitCode=0 Oct 30 17:39:15.719491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.719310 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" event={"ID":"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff","Type":"ContainerDied","Data":"85ae1bb8a04c850d5d0f8794619b2ba957c4026d5cdebed9724e5be91aae263f"} Oct 30 17:39:15.719491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.719384 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" event={"ID":"e5f0f53d-5f1a-4068-bea3-185bb56ab9ff","Type":"ContainerDied","Data":"e8ec082109be36e42d9bfad917cfe8a3dceb13669c3156060ca241c478f5da3d"} Oct 30 17:39:15.719491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.719321 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp" Oct 30 17:39:15.719491 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.719408 4828 scope.go:122] "RemoveContainer" containerID="85ae1bb8a04c850d5d0f8794619b2ba957c4026d5cdebed9724e5be91aae263f" Oct 30 17:39:15.730877 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.730859 4828 scope.go:122] "RemoveContainer" containerID="85ae1bb8a04c850d5d0f8794619b2ba957c4026d5cdebed9724e5be91aae263f" Oct 30 17:39:15.731194 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:39:15.731177 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ae1bb8a04c850d5d0f8794619b2ba957c4026d5cdebed9724e5be91aae263f\": container with ID starting with 85ae1bb8a04c850d5d0f8794619b2ba957c4026d5cdebed9724e5be91aae263f not found: ID does not exist" containerID="85ae1bb8a04c850d5d0f8794619b2ba957c4026d5cdebed9724e5be91aae263f" Oct 30 17:39:15.731240 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.731205 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ae1bb8a04c850d5d0f8794619b2ba957c4026d5cdebed9724e5be91aae263f"} err="failed to get container status \"85ae1bb8a04c850d5d0f8794619b2ba957c4026d5cdebed9724e5be91aae263f\": rpc error: code = NotFound desc = could not find container \"85ae1bb8a04c850d5d0f8794619b2ba957c4026d5cdebed9724e5be91aae263f\": container with ID starting with 85ae1bb8a04c850d5d0f8794619b2ba957c4026d5cdebed9724e5be91aae263f not found: ID does not exist" Oct 30 17:39:15.775109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.775008 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp"] Oct 30 17:39:15.790574 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.790530 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-096d75-db4fd7549-77gdp"] Oct 30 17:39:15.855919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:15.855872 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:16.143235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:16.143146 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:16.490360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:16.490283 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f0f53d-5f1a-4068-bea3-185bb56ab9ff" path="/var/lib/kubelet/pods/e5f0f53d-5f1a-4068-bea3-185bb56ab9ff/volumes" Oct 30 17:39:18.117396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:18.117358 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:19.853781 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:19.853717 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:21.645356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.645200 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:21.733046 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.733019 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" Oct 30 17:39:21.768085 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.768048 4828 generic.go:356] "Generic (PLEG): container finished" podID="cf528e56-af27-46d6-8f3d-20cc44101152" containerID="d4bb4bd6bc1d280252c085b76de47cb11c2c372c86dee8e90827733b949c97d4" exitCode=0 Oct 30 17:39:21.768261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.768104 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" Oct 30 17:39:21.768261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.768122 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" event={"ID":"cf528e56-af27-46d6-8f3d-20cc44101152","Type":"ContainerDied","Data":"d4bb4bd6bc1d280252c085b76de47cb11c2c372c86dee8e90827733b949c97d4"} Oct 30 17:39:21.768261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.768202 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45" event={"ID":"cf528e56-af27-46d6-8f3d-20cc44101152","Type":"ContainerDied","Data":"90c1acd243e0f2d24943c7b8d41768afff8d235834eeac9b5f51c9e01c534d6e"} Oct 30 17:39:21.768261 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.768227 4828 scope.go:122] "RemoveContainer" containerID="d4bb4bd6bc1d280252c085b76de47cb11c2c372c86dee8e90827733b949c97d4" Oct 30 17:39:21.780851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.780818 4828 scope.go:122] "RemoveContainer" containerID="d4bb4bd6bc1d280252c085b76de47cb11c2c372c86dee8e90827733b949c97d4" Oct 30 17:39:21.781671 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:39:21.781646 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4bb4bd6bc1d280252c085b76de47cb11c2c372c86dee8e90827733b949c97d4\": container with ID starting with d4bb4bd6bc1d280252c085b76de47cb11c2c372c86dee8e90827733b949c97d4 not found: ID does not exist" containerID="d4bb4bd6bc1d280252c085b76de47cb11c2c372c86dee8e90827733b949c97d4" Oct 30 17:39:21.781794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.781683 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4bb4bd6bc1d280252c085b76de47cb11c2c372c86dee8e90827733b949c97d4"} err="failed to get container status \"d4bb4bd6bc1d280252c085b76de47cb11c2c372c86dee8e90827733b949c97d4\": rpc error: code = NotFound desc = could not find container \"d4bb4bd6bc1d280252c085b76de47cb11c2c372c86dee8e90827733b949c97d4\": container with ID starting with d4bb4bd6bc1d280252c085b76de47cb11c2c372c86dee8e90827733b949c97d4 not found: ID does not exist" Oct 30 17:39:21.799738 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.799701 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf528e56-af27-46d6-8f3d-20cc44101152-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf528e56-af27-46d6-8f3d-20cc44101152-plugins-config-volume\") pod \"cf528e56-af27-46d6-8f3d-20cc44101152\" (UID: \"cf528e56-af27-46d6-8f3d-20cc44101152\") " Oct 30 17:39:21.799980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.799882 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf528e56-af27-46d6-8f3d-20cc44101152-kube-api-access-hdph2\" (UniqueName: \"kubernetes.io/projected/cf528e56-af27-46d6-8f3d-20cc44101152-kube-api-access-hdph2\") pod \"cf528e56-af27-46d6-8f3d-20cc44101152\" (UID: \"cf528e56-af27-46d6-8f3d-20cc44101152\") " Oct 30 17:39:21.800275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.800241 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf528e56-af27-46d6-8f3d-20cc44101152-plugins-config-volume" pod "cf528e56-af27-46d6-8f3d-20cc44101152" (UID: "cf528e56-af27-46d6-8f3d-20cc44101152"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:21.802552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.802527 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf528e56-af27-46d6-8f3d-20cc44101152-kube-api-access-hdph2" pod "cf528e56-af27-46d6-8f3d-20cc44101152" (UID: "cf528e56-af27-46d6-8f3d-20cc44101152"). InnerVolumeSpecName "kube-api-access-hdph2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:21.901308 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.901263 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf528e56-af27-46d6-8f3d-20cc44101152-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:21.901308 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:21.901301 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdph2\" (UniqueName: \"kubernetes.io/projected/cf528e56-af27-46d6-8f3d-20cc44101152-kube-api-access-hdph2\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:22.118947 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:22.118890 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45"] Oct 30 17:39:22.129151 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:22.129120 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-096d75-796b4f7cb7-7mn45"] Oct 30 17:39:22.492695 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:22.492619 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf528e56-af27-46d6-8f3d-20cc44101152" path="/var/lib/kubelet/pods/cf528e56-af27-46d6-8f3d-20cc44101152/volumes" Oct 30 17:39:22.952818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:22.952773 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:23.105440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:23.105404 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:23.243044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:23.242968 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g"] Oct 30 17:39:23.243228 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:23.243167 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" podUID="aab33acc-67cf-4cc4-b180-50cdee10b872" containerName="istio-proxy" containerID="cri-o://f4ed8258ac28f87b1f355565bdd0e6b6e254f011779336a54bc7df94510834c2" gracePeriod=30 Oct 30 17:39:23.278218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:23.278184 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:23.438478 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:23.438435 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:24.957100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:24.957054 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:27.297027 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:27.296984 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:28.211350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.211307 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:28.489159 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.488829 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:28.546019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.545993 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:39:28.678458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.678419 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-credential-socket\") pod \"aab33acc-67cf-4cc4-b180-50cdee10b872\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " Oct 30 17:39:28.678691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.678496 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-certs\") pod \"aab33acc-67cf-4cc4-b180-50cdee10b872\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " Oct 30 17:39:28.678691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.678527 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-token\" (UniqueName: \"kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-token\") pod \"aab33acc-67cf-4cc4-b180-50cdee10b872\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " Oct 30 17:39:28.678691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.678580 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-podinfo\") pod \"aab33acc-67cf-4cc4-b180-50cdee10b872\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " Oct 30 17:39:28.678691 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.678635 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-kube-api-access-hhfwk\" (UniqueName: \"kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-kube-api-access-hhfwk\") pod \"aab33acc-67cf-4cc4-b180-50cdee10b872\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " Oct 30 17:39:28.679013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.678725 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-socket\") pod \"aab33acc-67cf-4cc4-b180-50cdee10b872\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " Oct 30 17:39:28.679013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.678768 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aab33acc-67cf-4cc4-b180-50cdee10b872-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab33acc-67cf-4cc4-b180-50cdee10b872-istiod-ca-cert\") pod \"aab33acc-67cf-4cc4-b180-50cdee10b872\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " Oct 30 17:39:28.679013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.678828 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-credential-socket" pod "aab33acc-67cf-4cc4-b180-50cdee10b872" (UID: "aab33acc-67cf-4cc4-b180-50cdee10b872"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:28.679013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.678844 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-data\") pod \"aab33acc-67cf-4cc4-b180-50cdee10b872\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " Oct 30 17:39:28.679013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.678986 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-envoy\") pod \"aab33acc-67cf-4cc4-b180-50cdee10b872\" (UID: \"aab33acc-67cf-4cc4-b180-50cdee10b872\") " Oct 30 17:39:28.679375 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.679092 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-data" pod "aab33acc-67cf-4cc4-b180-50cdee10b872" (UID: "aab33acc-67cf-4cc4-b180-50cdee10b872"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:28.679375 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.679206 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-socket" pod "aab33acc-67cf-4cc4-b180-50cdee10b872" (UID: "aab33acc-67cf-4cc4-b180-50cdee10b872"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:28.679490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.679373 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab33acc-67cf-4cc4-b180-50cdee10b872-istiod-ca-cert" pod "aab33acc-67cf-4cc4-b180-50cdee10b872" (UID: "aab33acc-67cf-4cc4-b180-50cdee10b872"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:39:28.679570 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.679550 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.679627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.679576 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.679627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.679593 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab33acc-67cf-4cc4-b180-50cdee10b872-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.679627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.679607 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.679962 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.679941 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-certs" pod "aab33acc-67cf-4cc4-b180-50cdee10b872" (UID: "aab33acc-67cf-4cc4-b180-50cdee10b872"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:28.681552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.681517 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-token" pod "aab33acc-67cf-4cc4-b180-50cdee10b872" (UID: "aab33acc-67cf-4cc4-b180-50cdee10b872"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:28.681872 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.681853 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-envoy" pod "aab33acc-67cf-4cc4-b180-50cdee10b872" (UID: "aab33acc-67cf-4cc4-b180-50cdee10b872"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:39:28.682181 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.682162 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-kube-api-access-hhfwk" pod "aab33acc-67cf-4cc4-b180-50cdee10b872" (UID: "aab33acc-67cf-4cc4-b180-50cdee10b872"). InnerVolumeSpecName "kube-api-access-hhfwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:39:28.682650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.682633 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-podinfo" pod "aab33acc-67cf-4cc4-b180-50cdee10b872" (UID: "aab33acc-67cf-4cc4-b180-50cdee10b872"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:39:28.751759 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.751644 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:28.781365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.781323 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.781365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.781367 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab33acc-67cf-4cc4-b180-50cdee10b872-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.781576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.781382 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.781576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.781397 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab33acc-67cf-4cc4-b180-50cdee10b872-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.781576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.781413 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhfwk\" (UniqueName: \"kubernetes.io/projected/aab33acc-67cf-4cc4-b180-50cdee10b872-kube-api-access-hhfwk\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:39:28.816037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.816006 4828 generic.go:356] "Generic (PLEG): container finished" podID="aab33acc-67cf-4cc4-b180-50cdee10b872" containerID="f4ed8258ac28f87b1f355565bdd0e6b6e254f011779336a54bc7df94510834c2" exitCode=0 Oct 30 17:39:28.816210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.816073 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" Oct 30 17:39:28.816210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.816097 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" event={"ID":"aab33acc-67cf-4cc4-b180-50cdee10b872","Type":"ContainerDied","Data":"f4ed8258ac28f87b1f355565bdd0e6b6e254f011779336a54bc7df94510834c2"} Oct 30 17:39:28.816210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.816151 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g" event={"ID":"aab33acc-67cf-4cc4-b180-50cdee10b872","Type":"ContainerDied","Data":"081061cca22f7451f23b3c725fa6849b150274aaf1e02bb7a818c68427444140"} Oct 30 17:39:28.816210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.816177 4828 scope.go:122] "RemoveContainer" containerID="f4ed8258ac28f87b1f355565bdd0e6b6e254f011779336a54bc7df94510834c2" Oct 30 17:39:28.830059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.830007 4828 scope.go:122] "RemoveContainer" containerID="f4ed8258ac28f87b1f355565bdd0e6b6e254f011779336a54bc7df94510834c2" Oct 30 17:39:28.830488 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:39:28.830460 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ed8258ac28f87b1f355565bdd0e6b6e254f011779336a54bc7df94510834c2\": container with ID starting with f4ed8258ac28f87b1f355565bdd0e6b6e254f011779336a54bc7df94510834c2 not found: ID does not exist" containerID="f4ed8258ac28f87b1f355565bdd0e6b6e254f011779336a54bc7df94510834c2" Oct 30 17:39:28.830629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.830496 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ed8258ac28f87b1f355565bdd0e6b6e254f011779336a54bc7df94510834c2"} err="failed to get container status \"f4ed8258ac28f87b1f355565bdd0e6b6e254f011779336a54bc7df94510834c2\": rpc error: code = NotFound desc = could not find container \"f4ed8258ac28f87b1f355565bdd0e6b6e254f011779336a54bc7df94510834c2\": container with ID starting with f4ed8258ac28f87b1f355565bdd0e6b6e254f011779336a54bc7df94510834c2 not found: ID does not exist" Oct 30 17:39:28.876444 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.876399 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g"] Oct 30 17:39:28.885102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.885068 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-096d75-openshift-default-b9765b679-9w29g"] Oct 30 17:39:28.976988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:28.976947 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:29.136198 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:29.136161 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:30.166519 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:30.166475 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:30.488792 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:30.488704 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab33acc-67cf-4cc4-b180-50cdee10b872" path="/var/lib/kubelet/pods/aab33acc-67cf-4cc4-b180-50cdee10b872/volumes" Oct 30 17:39:30.739654 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:30.739131 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:32.281177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:32.280632 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:33.878827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:33.878721 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:35.996893 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:35.996834 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:36.601727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:36.601686 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:37.533385 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:37.533350 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:37.911172 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:39:37.911088 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:39:38.151971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:38.151924 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:38.298786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:38.298748 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:39.269648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:39.269531 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:40.792696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:40.792657 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:42.472044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:42.469684 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:44.587878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:44.587832 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:46.216892 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:46.216852 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:47.879842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:47.879803 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:49.143951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:49.143623 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:49.363762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:49.362681 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:49.715389 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:49.715352 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:51.090689 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:51.090634 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:53.177312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:53.177273 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:54.861184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:54.861097 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:56.282369 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:56.282341 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:56.478215 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:56.478178 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:56.565640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:56.565535 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf"] Oct 30 17:39:56.565838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:56.565786 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" containerID="cri-o://8fef74fd88446e6f699c459cc8a25bfcfad6758904ae65ea299fda89c545356b" gracePeriod=30 Oct 30 17:39:56.637291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:56.637242 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:58.118588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:58.118540 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:39:59.176732 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:59.176695 4828 patch_prober.go:29] interesting pod/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:39:59.177149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:59.176768 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:39:59.809865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:39:59.809825 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:00.656627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:00.656595 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:01.294215 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.294181 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:01.829604 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.829541 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:40:01.913319 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.913278 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:01.930395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.930360 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-podinfo\") pod \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " Oct 30 17:40:01.930585 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.930419 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-credential-socket\") pod \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " Oct 30 17:40:01.930585 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.930441 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-kube-api-access-vf6bb\" (UniqueName: \"kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-kube-api-access-vf6bb\") pod \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " Oct 30 17:40:01.930585 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.930475 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-certs\") pod \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " Oct 30 17:40:01.930585 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.930527 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-token\" (UniqueName: \"kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-token\") pod \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " Oct 30 17:40:01.930819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.930587 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-envoy\") pod \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " Oct 30 17:40:01.930819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.930623 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-socket\") pod \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " Oct 30 17:40:01.930819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.930687 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-data\") pod \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " Oct 30 17:40:01.930819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.930744 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istiod-ca-cert\") pod \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\" (UID: \"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5\") " Oct 30 17:40:01.930819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.930802 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-certs" pod "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" (UID: "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:01.931095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.930860 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-credential-socket" pod "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" (UID: "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:01.931226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.931169 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:01.931226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.931196 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:01.931226 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.931212 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istiod-ca-cert" pod "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" (UID: "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:01.931429 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.931350 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-socket" pod "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" (UID: "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:01.931429 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.931399 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-data" pod "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" (UID: "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:01.934788 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.934763 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-envoy" pod "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" (UID: "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:01.934917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.934815 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-kube-api-access-vf6bb" pod "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" (UID: "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5"). InnerVolumeSpecName "kube-api-access-vf6bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:01.934917 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.934837 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-podinfo" pod "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" (UID: "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:40:01.935832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:01.935811 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-token" pod "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" (UID: "a5afa0e3-1eda-4826-bcf7-1225ab2e50f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:02.014390 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.014353 4828 generic.go:356] "Generic (PLEG): container finished" podID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerID="8fef74fd88446e6f699c459cc8a25bfcfad6758904ae65ea299fda89c545356b" exitCode=0 Oct 30 17:40:02.014588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.014408 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" Oct 30 17:40:02.014588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.014429 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" event={"ID":"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5","Type":"ContainerDied","Data":"8fef74fd88446e6f699c459cc8a25bfcfad6758904ae65ea299fda89c545356b"} Oct 30 17:40:02.014588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.014483 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf" event={"ID":"a5afa0e3-1eda-4826-bcf7-1225ab2e50f5","Type":"ContainerDied","Data":"056dc9681adc5fb3e738225b3a78859d36b46bbef362bcf280b437ebb95de278"} Oct 30 17:40:02.014588 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.014504 4828 scope.go:122] "RemoveContainer" containerID="8fef74fd88446e6f699c459cc8a25bfcfad6758904ae65ea299fda89c545356b" Oct 30 17:40:02.024687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.024666 4828 scope.go:122] "RemoveContainer" containerID="fa94b91391edcffb19b35439803dad5a28e1676757939c06affb8413468de3a8" Oct 30 17:40:02.032612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.032590 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:02.032612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.032612 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:02.032750 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.032622 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:02.032750 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.032631 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:02.032750 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.032641 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:02.032750 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.032649 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:02.032750 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.032657 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf6bb\" (UniqueName: \"kubernetes.io/projected/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5-kube-api-access-vf6bb\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:02.033865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.033853 4828 scope.go:122] "RemoveContainer" containerID="8fef74fd88446e6f699c459cc8a25bfcfad6758904ae65ea299fda89c545356b" Oct 30 17:40:02.034151 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:40:02.034133 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fef74fd88446e6f699c459cc8a25bfcfad6758904ae65ea299fda89c545356b\": container with ID starting with 8fef74fd88446e6f699c459cc8a25bfcfad6758904ae65ea299fda89c545356b not found: ID does not exist" containerID="8fef74fd88446e6f699c459cc8a25bfcfad6758904ae65ea299fda89c545356b" Oct 30 17:40:02.034200 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.034159 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fef74fd88446e6f699c459cc8a25bfcfad6758904ae65ea299fda89c545356b"} err="failed to get container status \"8fef74fd88446e6f699c459cc8a25bfcfad6758904ae65ea299fda89c545356b\": rpc error: code = NotFound desc = could not find container \"8fef74fd88446e6f699c459cc8a25bfcfad6758904ae65ea299fda89c545356b\": container with ID starting with 8fef74fd88446e6f699c459cc8a25bfcfad6758904ae65ea299fda89c545356b not found: ID does not exist" Oct 30 17:40:02.034200 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.034174 4828 scope.go:122] "RemoveContainer" containerID="fa94b91391edcffb19b35439803dad5a28e1676757939c06affb8413468de3a8" Oct 30 17:40:02.034380 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:40:02.034367 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa94b91391edcffb19b35439803dad5a28e1676757939c06affb8413468de3a8\": container with ID starting with fa94b91391edcffb19b35439803dad5a28e1676757939c06affb8413468de3a8 not found: ID does not exist" containerID="fa94b91391edcffb19b35439803dad5a28e1676757939c06affb8413468de3a8" Oct 30 17:40:02.034421 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.034382 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa94b91391edcffb19b35439803dad5a28e1676757939c06affb8413468de3a8"} err="failed to get container status \"fa94b91391edcffb19b35439803dad5a28e1676757939c06affb8413468de3a8\": rpc error: code = NotFound desc = could not find container \"fa94b91391edcffb19b35439803dad5a28e1676757939c06affb8413468de3a8\": container with ID starting with fa94b91391edcffb19b35439803dad5a28e1676757939c06affb8413468de3a8 not found: ID does not exist" Oct 30 17:40:02.069220 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.069187 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf"] Oct 30 17:40:02.075440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.075414 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-8f92ef-openshift-default-6c89cfc6f9-kmgvf"] Oct 30 17:40:02.489667 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:02.489630 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" path="/var/lib/kubelet/pods/a5afa0e3-1eda-4826-bcf7-1225ab2e50f5/volumes" Oct 30 17:40:03.694321 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:03.694226 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:05.320991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:05.320890 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:07.021503 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:07.021462 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:07.910000 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:40:07.909964 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:40:08.245786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.245694 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg"] Oct 30 17:40:08.246275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.245951 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" podUID="95d6e315-e9a3-47d1-a4b3-61b43c3d611b" containerName="maas-api" containerID="cri-o://3376a24c250f96f22d3d5416b5bbcfecb63e6cce1032c3a3202e00d6ba318892" gracePeriod=30 Oct 30 17:40:08.419319 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.419247 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:08.527911 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.527873 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:40:08.600173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.600137 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-openshift-service-ca\") pod \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " Oct 30 17:40:08.600395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.600265 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-metrics-certs\") pod \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " Oct 30 17:40:08.600395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.600305 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-kube-api-access-4mgt2\" (UniqueName: \"kubernetes.io/projected/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-kube-api-access-4mgt2\") pod \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " Oct 30 17:40:08.600395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.600387 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-maas-api-tls\") pod \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\" (UID: \"95d6e315-e9a3-47d1-a4b3-61b43c3d611b\") " Oct 30 17:40:08.600591 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.600570 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-openshift-service-ca" pod "95d6e315-e9a3-47d1-a4b3-61b43c3d611b" (UID: "95d6e315-e9a3-47d1-a4b3-61b43c3d611b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:08.600827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.600809 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:08.602768 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.602731 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-maas-api-tls" pod "95d6e315-e9a3-47d1-a4b3-61b43c3d611b" (UID: "95d6e315-e9a3-47d1-a4b3-61b43c3d611b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:40:08.603080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.603057 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-kube-api-access-4mgt2" pod "95d6e315-e9a3-47d1-a4b3-61b43c3d611b" (UID: "95d6e315-e9a3-47d1-a4b3-61b43c3d611b"). InnerVolumeSpecName "kube-api-access-4mgt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:08.603149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.603103 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-metrics-certs" pod "95d6e315-e9a3-47d1-a4b3-61b43c3d611b" (UID: "95d6e315-e9a3-47d1-a4b3-61b43c3d611b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:40:08.652801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.652761 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:08.702291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.702255 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:08.702291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.702286 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mgt2\" (UniqueName: \"kubernetes.io/projected/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-kube-api-access-4mgt2\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:08.702291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.702296 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95d6e315-e9a3-47d1-a4b3-61b43c3d611b-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:08.844217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:08.843750 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:09.031297 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:09.031257 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:09.065799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:09.065761 4828 generic.go:356] "Generic (PLEG): container finished" podID="95d6e315-e9a3-47d1-a4b3-61b43c3d611b" containerID="3376a24c250f96f22d3d5416b5bbcfecb63e6cce1032c3a3202e00d6ba318892" exitCode=0 Oct 30 17:40:09.065991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:09.065829 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" Oct 30 17:40:09.065991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:09.065847 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" event={"ID":"95d6e315-e9a3-47d1-a4b3-61b43c3d611b","Type":"ContainerDied","Data":"3376a24c250f96f22d3d5416b5bbcfecb63e6cce1032c3a3202e00d6ba318892"} Oct 30 17:40:09.065991 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:09.065969 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg" event={"ID":"95d6e315-e9a3-47d1-a4b3-61b43c3d611b","Type":"ContainerDied","Data":"598ab3d4c44fbcb2da711969b4ce8000c5b7e59c6368622c43ffc610e8fa5a25"} Oct 30 17:40:09.066113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:09.065994 4828 scope.go:122] "RemoveContainer" containerID="3376a24c250f96f22d3d5416b5bbcfecb63e6cce1032c3a3202e00d6ba318892" Oct 30 17:40:09.077214 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:09.077189 4828 scope.go:122] "RemoveContainer" containerID="3376a24c250f96f22d3d5416b5bbcfecb63e6cce1032c3a3202e00d6ba318892" Oct 30 17:40:09.077498 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:40:09.077476 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3376a24c250f96f22d3d5416b5bbcfecb63e6cce1032c3a3202e00d6ba318892\": container with ID starting with 3376a24c250f96f22d3d5416b5bbcfecb63e6cce1032c3a3202e00d6ba318892 not found: ID does not exist" containerID="3376a24c250f96f22d3d5416b5bbcfecb63e6cce1032c3a3202e00d6ba318892" Oct 30 17:40:09.077558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:09.077507 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3376a24c250f96f22d3d5416b5bbcfecb63e6cce1032c3a3202e00d6ba318892"} err="failed to get container status \"3376a24c250f96f22d3d5416b5bbcfecb63e6cce1032c3a3202e00d6ba318892\": rpc error: code = NotFound desc = could not find container \"3376a24c250f96f22d3d5416b5bbcfecb63e6cce1032c3a3202e00d6ba318892\": container with ID starting with 3376a24c250f96f22d3d5416b5bbcfecb63e6cce1032c3a3202e00d6ba318892 not found: ID does not exist" Oct 30 17:40:09.121366 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:09.121269 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg"] Oct 30 17:40:09.133370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:09.133328 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-054acd-5cff8b97-2grzg"] Oct 30 17:40:09.183681 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:09.183640 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:09.681047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:09.681004 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:10.489078 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:10.489040 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d6e315-e9a3-47d1-a4b3-61b43c3d611b" path="/var/lib/kubelet/pods/95d6e315-e9a3-47d1-a4b3-61b43c3d611b/volumes" Oct 30 17:40:10.948267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:10.948231 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:12.529056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:12.529015 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:14.149032 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:14.148997 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:15.673481 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:15.673401 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:16.059650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.059605 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:16.206184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.206150 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:16.336951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.336851 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt"] Oct 30 17:40:16.337201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.337158 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerName="istio-proxy" containerID="cri-o://7c0b95184ad45c10af27ce58166e660281099b693d3078618d4e130fe761f864" gracePeriod=30 Oct 30 17:40:16.414539 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.414487 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:16.897911 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.897867 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6"] Oct 30 17:40:16.898581 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898560 4828 cpu_manager.go:414] "containerMap: removing container" podUID="1b3a84af-dd51-40b7-b426-b51d208ca058" containerName="payload-pre-processing" Oct 30 17:40:16.898635 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898585 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3a84af-dd51-40b7-b426-b51d208ca058" containerName="payload-pre-processing" Oct 30 17:40:16.898635 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898603 4828 cpu_manager.go:414] "containerMap: removing container" podUID="264f7990-6cc2-4018-a1c5-cc76ccfb5338" containerName="maas-api" Oct 30 17:40:16.898635 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898611 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264f7990-6cc2-4018-a1c5-cc76ccfb5338" containerName="maas-api" Oct 30 17:40:16.898635 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898631 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c64dc903-607a-4627-92a7-74f18599c40b" containerName="payload-processing" Oct 30 17:40:16.898764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898639 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64dc903-607a-4627-92a7-74f18599c40b" containerName="payload-processing" Oct 30 17:40:16.898764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898655 4828 cpu_manager.go:414] "containerMap: removing container" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" Oct 30 17:40:16.898764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898662 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" Oct 30 17:40:16.898764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898671 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c4eb9078-46ec-4fa4-b0a6-dc78e060859e" containerName="payload-pre-processing" Oct 30 17:40:16.898764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898678 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4eb9078-46ec-4fa4-b0a6-dc78e060859e" containerName="payload-pre-processing" Oct 30 17:40:16.898764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898690 4828 cpu_manager.go:414] "containerMap: removing container" podUID="5eb25a1e-9f6b-4823-982a-82b25909fd53" containerName="istio-proxy" Oct 30 17:40:16.898764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898698 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb25a1e-9f6b-4823-982a-82b25909fd53" containerName="istio-proxy" Oct 30 17:40:16.898764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898710 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" Oct 30 17:40:16.898764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898717 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" Oct 30 17:40:16.898764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898727 4828 cpu_manager.go:414] "containerMap: removing container" podUID="95d6e315-e9a3-47d1-a4b3-61b43c3d611b" containerName="maas-api" Oct 30 17:40:16.898764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898735 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d6e315-e9a3-47d1-a4b3-61b43c3d611b" containerName="maas-api" Oct 30 17:40:16.898764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898750 4828 cpu_manager.go:414] "containerMap: removing container" podUID="606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a" containerName="maas-api" Oct 30 17:40:16.898764 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898757 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a" containerName="maas-api" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898768 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898775 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898794 4828 cpu_manager.go:414] "containerMap: removing container" podUID="52e9873d-a365-4097-af99-34bca8251f98" containerName="payload-pre-processing" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898801 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e9873d-a365-4097-af99-34bca8251f98" containerName="payload-pre-processing" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898812 4828 cpu_manager.go:414] "containerMap: removing container" podUID="d0ee8b49-7ba2-485a-8f44-92f365ea306f" containerName="maas-api" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898820 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ee8b49-7ba2-485a-8f44-92f365ea306f" containerName="maas-api" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898830 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898837 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898852 4828 cpu_manager.go:414] "containerMap: removing container" podUID="cf528e56-af27-46d6-8f3d-20cc44101152" containerName="payload-pre-processing" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898859 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf528e56-af27-46d6-8f3d-20cc44101152" containerName="payload-pre-processing" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898871 4828 cpu_manager.go:414] "containerMap: removing container" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898878 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898887 4828 cpu_manager.go:414] "containerMap: removing container" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898894 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898919 4828 cpu_manager.go:414] "containerMap: removing container" podUID="66bda2af-c577-42eb-bfdf-1b63d8433886" containerName="maas-api" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898925 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66bda2af-c577-42eb-bfdf-1b63d8433886" containerName="maas-api" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898936 4828 cpu_manager.go:414] "containerMap: removing container" podUID="07e31526-445e-4541-8026-e134539da213" containerName="maas-api" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898943 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e31526-445e-4541-8026-e134539da213" containerName="maas-api" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898953 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898959 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898971 4828 cpu_manager.go:414] "containerMap: removing container" podUID="dd6e82e3-8647-478a-b1d9-9be624746f21" containerName="payload-processing" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898978 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6e82e3-8647-478a-b1d9-9be624746f21" containerName="payload-processing" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898991 4828 cpu_manager.go:414] "containerMap: removing container" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.898998 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899007 4828 cpu_manager.go:414] "containerMap: removing container" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899014 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899025 4828 cpu_manager.go:414] "containerMap: removing container" podUID="f92a7745-1da0-468d-bbb0-fba201967378" containerName="payload-processing" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899032 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f92a7745-1da0-468d-bbb0-fba201967378" containerName="payload-processing" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899043 4828 cpu_manager.go:414] "containerMap: removing container" podUID="666617d1-e514-4b17-ae65-1da9dfeea316" containerName="payload-processing" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899050 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666617d1-e514-4b17-ae65-1da9dfeea316" containerName="payload-processing" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899062 4828 cpu_manager.go:414] "containerMap: removing container" podUID="93b9082e-ac02-4490-a9aa-d805d7f9d737" containerName="maas-api" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899070 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93b9082e-ac02-4490-a9aa-d805d7f9d737" containerName="maas-api" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899085 4828 cpu_manager.go:414] "containerMap: removing container" podUID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899092 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899107 4828 cpu_manager.go:414] "containerMap: removing container" podUID="e5f0f53d-5f1a-4068-bea3-185bb56ab9ff" containerName="maas-api" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899113 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f0f53d-5f1a-4068-bea3-185bb56ab9ff" containerName="maas-api" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899123 4828 cpu_manager.go:414] "containerMap: removing container" podUID="14d8d98a-d3b2-4c79-9ebe-d9f817498f61" containerName="payload-pre-processing" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899130 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d8d98a-d3b2-4c79-9ebe-d9f817498f61" containerName="payload-pre-processing" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899144 4828 cpu_manager.go:414] "containerMap: removing container" podUID="aab33acc-67cf-4cc4-b180-50cdee10b872" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899152 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab33acc-67cf-4cc4-b180-50cdee10b872" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899163 4828 cpu_manager.go:414] "containerMap: removing container" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899170 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f119ba5-b435-453d-809e-bb2795e3066d" containerName="istio-proxy" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899182 4828 cpu_manager.go:414] "containerMap: removing container" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899189 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899200 4828 cpu_manager.go:414] "containerMap: removing container" podUID="9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58" containerName="maas-api" Oct 30 17:40:16.899193 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899207 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58" containerName="maas-api" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899341 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e9873d-a365-4097-af99-34bca8251f98" containerName="payload-pre-processing" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899356 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ee8b49-7ba2-485a-8f44-92f365ea306f" containerName="maas-api" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899365 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899376 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf528e56-af27-46d6-8f3d-20cc44101152" containerName="payload-pre-processing" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899388 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899397 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="31569569-b53b-43c8-a6ca-2970efc5e331" containerName="main" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899405 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e31526-445e-4541-8026-e134539da213" containerName="maas-api" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899415 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="66bda2af-c577-42eb-bfdf-1b63d8433886" containerName="maas-api" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899424 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899431 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6e82e3-8647-478a-b1d9-9be624746f21" containerName="payload-processing" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899440 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="05de1e06-0e9b-46a5-9020-5d791da7654a" containerName="istio-proxy" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899448 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899455 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="f92a7745-1da0-468d-bbb0-fba201967378" containerName="payload-processing" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899463 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="666617d1-e514-4b17-ae65-1da9dfeea316" containerName="payload-processing" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899471 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="93b9082e-ac02-4490-a9aa-d805d7f9d737" containerName="maas-api" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899479 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="3454fdaa-9f5d-40c0-9ce5-7a801a281024" containerName="istio-proxy" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899490 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f0f53d-5f1a-4068-bea3-185bb56ab9ff" containerName="maas-api" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899499 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d8d98a-d3b2-4c79-9ebe-d9f817498f61" containerName="payload-pre-processing" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899507 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab33acc-67cf-4cc4-b180-50cdee10b872" containerName="istio-proxy" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899515 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb327ed-88b8-4f18-81b7-881fc4480618" containerName="main" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899523 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f9b625a-fe9c-49b6-a2e0-22c1ac0a1e58" containerName="maas-api" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899533 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3a84af-dd51-40b7-b426-b51d208ca058" containerName="payload-pre-processing" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899542 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="264f7990-6cc2-4018-a1c5-cc76ccfb5338" containerName="maas-api" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899552 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64dc903-607a-4627-92a7-74f18599c40b" containerName="payload-processing" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899563 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a75163-e6e5-4eaa-9197-a0e16de86f19" containerName="istio-proxy" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899574 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4eb9078-46ec-4fa4-b0a6-dc78e060859e" containerName="payload-pre-processing" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899584 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb25a1e-9f6b-4823-982a-82b25909fd53" containerName="istio-proxy" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899592 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d6e315-e9a3-47d1-a4b3-61b43c3d611b" containerName="maas-api" Oct 30 17:40:16.901013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.899601 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="606ccbcf-d73f-4e1e-9dc2-63b217fdcf2a" containerName="maas-api" Oct 30 17:40:16.905343 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.905312 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" Oct 30 17:40:16.921692 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.921660 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6"] Oct 30 17:40:16.977140 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.977102 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:40:16.977312 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.977284 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:40:16.987831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.987792 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6\" (UID: \"cee757d8-b555-433a-a85b-d6d3bd4ca416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" Oct 30 17:40:16.988044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.987942 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6\" (UID: \"cee757d8-b555-433a-a85b-d6d3bd4ca416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" Oct 30 17:40:16.988044 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:16.988014 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z4rr\" (UniqueName: \"kubernetes.io/projected/cee757d8-b555-433a-a85b-d6d3bd4ca416-kube-api-access-6z4rr\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6\" (UID: \"cee757d8-b555-433a-a85b-d6d3bd4ca416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" Oct 30 17:40:17.017513 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:17.017477 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:17.089012 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:17.088966 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6\" (UID: \"cee757d8-b555-433a-a85b-d6d3bd4ca416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" Oct 30 17:40:17.089218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:17.089059 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6\" (UID: \"cee757d8-b555-433a-a85b-d6d3bd4ca416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" Oct 30 17:40:17.089218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:17.089111 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z4rr\" (UniqueName: \"kubernetes.io/projected/cee757d8-b555-433a-a85b-d6d3bd4ca416-kube-api-access-6z4rr\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6\" (UID: \"cee757d8-b555-433a-a85b-d6d3bd4ca416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" Oct 30 17:40:17.089967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:17.089934 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6\" (UID: \"cee757d8-b555-433a-a85b-d6d3bd4ca416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" Oct 30 17:40:17.092223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:17.092186 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6\" (UID: \"cee757d8-b555-433a-a85b-d6d3bd4ca416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" Oct 30 17:40:17.113697 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:17.113664 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z4rr\" (UniqueName: \"kubernetes.io/projected/cee757d8-b555-433a-a85b-d6d3bd4ca416-kube-api-access-6z4rr\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6\" (UID: \"cee757d8-b555-433a-a85b-d6d3bd4ca416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" Oct 30 17:40:17.224490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:17.224395 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" Oct 30 17:40:17.416762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:17.416713 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:17.532004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:17.531975 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6"] Oct 30 17:40:17.532588 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:40:17.532563 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee757d8_b555_433a_a85b_d6d3bd4ca416.slice/crio-6b3a4327a8130002d6da5fbde9b290b887c63b6544e23b974d36b6a75d05d697 WatchSource:0}: Error finding container 6b3a4327a8130002d6da5fbde9b290b887c63b6544e23b974d36b6a75d05d697: Status 404 returned error can't find the container with id 6b3a4327a8130002d6da5fbde9b290b887c63b6544e23b974d36b6a75d05d697 Oct 30 17:40:18.123562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:18.123455 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" event={"ID":"cee757d8-b555-433a-a85b-d6d3bd4ca416","Type":"ContainerStarted","Data":"7bb2bb7365fdb134ff053ca9885f5a6c3b515617a02d24a0253810d691494702"} Oct 30 17:40:18.123562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:18.123530 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" event={"ID":"cee757d8-b555-433a-a85b-d6d3bd4ca416","Type":"ContainerStarted","Data":"6b3a4327a8130002d6da5fbde9b290b887c63b6544e23b974d36b6a75d05d697"} Oct 30 17:40:18.167206 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:18.167114 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" podStartSLOduration=2.167091496 podStartE2EDuration="2.167091496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:40:18.165504394 +0000 UTC m=+3195.811014575" watchObservedRunningTime="2026-10-30 17:40:18.167091496 +0000 UTC m=+3195.812601677" Oct 30 17:40:19.130694 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:19.130663 4828 generic.go:356] "Generic (PLEG): container finished" podID="cee757d8-b555-433a-a85b-d6d3bd4ca416" containerID="7bb2bb7365fdb134ff053ca9885f5a6c3b515617a02d24a0253810d691494702" exitCode=0 Oct 30 17:40:19.131136 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:19.130706 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" event={"ID":"cee757d8-b555-433a-a85b-d6d3bd4ca416","Type":"ContainerDied","Data":"7bb2bb7365fdb134ff053ca9885f5a6c3b515617a02d24a0253810d691494702"} Oct 30 17:40:19.148302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:19.148243 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:20.279329 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:20.279294 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" Oct 30 17:40:20.425450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:20.425352 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-xks-ca\") pod \"cee757d8-b555-433a-a85b-d6d3bd4ca416\" (UID: \"cee757d8-b555-433a-a85b-d6d3bd4ca416\") " Oct 30 17:40:20.425450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:20.425420 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-openshift-ca\") pod \"cee757d8-b555-433a-a85b-d6d3bd4ca416\" (UID: \"cee757d8-b555-433a-a85b-d6d3bd4ca416\") " Oct 30 17:40:20.425645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:20.425521 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee757d8-b555-433a-a85b-d6d3bd4ca416-kube-api-access-6z4rr\" (UniqueName: \"kubernetes.io/projected/cee757d8-b555-433a-a85b-d6d3bd4ca416-kube-api-access-6z4rr\") pod \"cee757d8-b555-433a-a85b-d6d3bd4ca416\" (UID: \"cee757d8-b555-433a-a85b-d6d3bd4ca416\") " Oct 30 17:40:20.425812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:20.425791 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-openshift-ca" pod "cee757d8-b555-433a-a85b-d6d3bd4ca416" (UID: "cee757d8-b555-433a-a85b-d6d3bd4ca416"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:20.426007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:20.425988 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-openshift-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:20.427673 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:20.427645 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-xks-ca" pod "cee757d8-b555-433a-a85b-d6d3bd4ca416" (UID: "cee757d8-b555-433a-a85b-d6d3bd4ca416"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:40:20.427783 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:20.427707 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee757d8-b555-433a-a85b-d6d3bd4ca416-kube-api-access-6z4rr" pod "cee757d8-b555-433a-a85b-d6d3bd4ca416" (UID: "cee757d8-b555-433a-a85b-d6d3bd4ca416"). InnerVolumeSpecName "kube-api-access-6z4rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:20.526787 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:20.526751 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cee757d8-b555-433a-a85b-d6d3bd4ca416-maas-api-xks-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:20.526787 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:20.526783 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z4rr\" (UniqueName: \"kubernetes.io/projected/cee757d8-b555-433a-a85b-d6d3bd4ca416-kube-api-access-6z4rr\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:20.692174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:20.692054 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:21.143407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.143374 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" Oct 30 17:40:21.143622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.143383 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6" event={"ID":"cee757d8-b555-433a-a85b-d6d3bd4ca416","Type":"ContainerDied","Data":"6b3a4327a8130002d6da5fbde9b290b887c63b6544e23b974d36b6a75d05d697"} Oct 30 17:40:21.143622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.143521 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b3a4327a8130002d6da5fbde9b290b887c63b6544e23b974d36b6a75d05d697" Oct 30 17:40:21.597322 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.597291 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:40:21.739582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.739500 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-credential-socket\") pod \"a3660fa4-e79d-4624-a792-d60275ef4c43\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " Oct 30 17:40:21.739582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.739546 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-podinfo\") pod \"a3660fa4-e79d-4624-a792-d60275ef4c43\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " Oct 30 17:40:21.739582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.739578 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3660fa4-e79d-4624-a792-d60275ef4c43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3660fa4-e79d-4624-a792-d60275ef4c43-istiod-ca-cert\") pod \"a3660fa4-e79d-4624-a792-d60275ef4c43\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " Oct 30 17:40:21.739878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.739606 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-certs\") pod \"a3660fa4-e79d-4624-a792-d60275ef4c43\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " Oct 30 17:40:21.739878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.739622 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-envoy\") pod \"a3660fa4-e79d-4624-a792-d60275ef4c43\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " Oct 30 17:40:21.739878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.739657 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-data\") pod \"a3660fa4-e79d-4624-a792-d60275ef4c43\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " Oct 30 17:40:21.739878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.739684 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-kube-api-access-slt7j\" (UniqueName: \"kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-kube-api-access-slt7j\") pod \"a3660fa4-e79d-4624-a792-d60275ef4c43\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " Oct 30 17:40:21.739878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.739730 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-token\" (UniqueName: \"kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-token\") pod \"a3660fa4-e79d-4624-a792-d60275ef4c43\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " Oct 30 17:40:21.739878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.739806 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-socket\") pod \"a3660fa4-e79d-4624-a792-d60275ef4c43\" (UID: \"a3660fa4-e79d-4624-a792-d60275ef4c43\") " Oct 30 17:40:21.739878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.739829 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-credential-socket" pod "a3660fa4-e79d-4624-a792-d60275ef4c43" (UID: "a3660fa4-e79d-4624-a792-d60275ef4c43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:21.740320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.740032 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-certs" pod "a3660fa4-e79d-4624-a792-d60275ef4c43" (UID: "a3660fa4-e79d-4624-a792-d60275ef4c43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:21.740320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.740081 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:21.740320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.740205 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-data" pod "a3660fa4-e79d-4624-a792-d60275ef4c43" (UID: "a3660fa4-e79d-4624-a792-d60275ef4c43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:21.740320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.740238 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3660fa4-e79d-4624-a792-d60275ef4c43-istiod-ca-cert" pod "a3660fa4-e79d-4624-a792-d60275ef4c43" (UID: "a3660fa4-e79d-4624-a792-d60275ef4c43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:21.740320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.740302 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-socket" pod "a3660fa4-e79d-4624-a792-d60275ef4c43" (UID: "a3660fa4-e79d-4624-a792-d60275ef4c43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:21.742049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.742023 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-envoy" pod "a3660fa4-e79d-4624-a792-d60275ef4c43" (UID: "a3660fa4-e79d-4624-a792-d60275ef4c43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:21.742157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.742140 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-podinfo" pod "a3660fa4-e79d-4624-a792-d60275ef4c43" (UID: "a3660fa4-e79d-4624-a792-d60275ef4c43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:40:21.742212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.742174 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-kube-api-access-slt7j" pod "a3660fa4-e79d-4624-a792-d60275ef4c43" (UID: "a3660fa4-e79d-4624-a792-d60275ef4c43"). InnerVolumeSpecName "kube-api-access-slt7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:21.742212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.742192 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-token" pod "a3660fa4-e79d-4624-a792-d60275ef4c43" (UID: "a3660fa4-e79d-4624-a792-d60275ef4c43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:21.841073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.841041 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:21.841073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.841068 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:21.841073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.841079 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3660fa4-e79d-4624-a792-d60275ef4c43-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:21.841317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.841089 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:21.841317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.841097 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:21.841317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.841105 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:21.841317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.841114 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slt7j\" (UniqueName: \"kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-kube-api-access-slt7j\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:21.841317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:21.841123 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3660fa4-e79d-4624-a792-d60275ef4c43-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:22.152120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.152082 4828 generic.go:356] "Generic (PLEG): container finished" podID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerID="7c0b95184ad45c10af27ce58166e660281099b693d3078618d4e130fe761f864" exitCode=0 Oct 30 17:40:22.152306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.152142 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" Oct 30 17:40:22.152306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.152167 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" event={"ID":"a3660fa4-e79d-4624-a792-d60275ef4c43","Type":"ContainerDied","Data":"7c0b95184ad45c10af27ce58166e660281099b693d3078618d4e130fe761f864"} Oct 30 17:40:22.152306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.152238 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt" event={"ID":"a3660fa4-e79d-4624-a792-d60275ef4c43","Type":"ContainerDied","Data":"2d7dbcddabbd8d2fbeefaada34ad414db8be741866079e01cb558fdb663b7684"} Oct 30 17:40:22.152306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.152267 4828 scope.go:122] "RemoveContainer" containerID="7c0b95184ad45c10af27ce58166e660281099b693d3078618d4e130fe761f864" Oct 30 17:40:22.162746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.162724 4828 scope.go:122] "RemoveContainer" containerID="503d48ce2722fd90e8109e1a7907d3b4bd73fc0451a7875047a1961b6b97b6bb" Oct 30 17:40:22.172580 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.172557 4828 scope.go:122] "RemoveContainer" containerID="7c0b95184ad45c10af27ce58166e660281099b693d3078618d4e130fe761f864" Oct 30 17:40:22.173010 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:40:22.172985 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c0b95184ad45c10af27ce58166e660281099b693d3078618d4e130fe761f864\": container with ID starting with 7c0b95184ad45c10af27ce58166e660281099b693d3078618d4e130fe761f864 not found: ID does not exist" containerID="7c0b95184ad45c10af27ce58166e660281099b693d3078618d4e130fe761f864" Oct 30 17:40:22.173115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.173025 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c0b95184ad45c10af27ce58166e660281099b693d3078618d4e130fe761f864"} err="failed to get container status \"7c0b95184ad45c10af27ce58166e660281099b693d3078618d4e130fe761f864\": rpc error: code = NotFound desc = could not find container \"7c0b95184ad45c10af27ce58166e660281099b693d3078618d4e130fe761f864\": container with ID starting with 7c0b95184ad45c10af27ce58166e660281099b693d3078618d4e130fe761f864 not found: ID does not exist" Oct 30 17:40:22.173115 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.173050 4828 scope.go:122] "RemoveContainer" containerID="503d48ce2722fd90e8109e1a7907d3b4bd73fc0451a7875047a1961b6b97b6bb" Oct 30 17:40:22.173357 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:40:22.173335 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503d48ce2722fd90e8109e1a7907d3b4bd73fc0451a7875047a1961b6b97b6bb\": container with ID starting with 503d48ce2722fd90e8109e1a7907d3b4bd73fc0451a7875047a1961b6b97b6bb not found: ID does not exist" containerID="503d48ce2722fd90e8109e1a7907d3b4bd73fc0451a7875047a1961b6b97b6bb" Oct 30 17:40:22.173408 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.173368 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503d48ce2722fd90e8109e1a7907d3b4bd73fc0451a7875047a1961b6b97b6bb"} err="failed to get container status \"503d48ce2722fd90e8109e1a7907d3b4bd73fc0451a7875047a1961b6b97b6bb\": rpc error: code = NotFound desc = could not find container \"503d48ce2722fd90e8109e1a7907d3b4bd73fc0451a7875047a1961b6b97b6bb\": container with ID starting with 503d48ce2722fd90e8109e1a7907d3b4bd73fc0451a7875047a1961b6b97b6bb not found: ID does not exist" Oct 30 17:40:22.209165 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.209122 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt"] Oct 30 17:40:22.213192 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.213155 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-054acd-openshift-default-5b8655cd47-zzjwt"] Oct 30 17:40:22.271367 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.271331 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:22.489596 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:22.489517 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" path="/var/lib/kubelet/pods/a3660fa4-e79d-4624-a792-d60275ef4c43/volumes" Oct 30 17:40:24.056881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:24.056837 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:25.690346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:25.689315 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:27.102796 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:27.102128 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:27.261796 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:27.261760 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:27.849860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:27.849774 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:28.436884 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:28.436846 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:28.508225 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:28.508193 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6"] Oct 30 17:40:28.524063 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:28.524023 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b87f3c-l7lw6"] Oct 30 17:40:28.604451 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:28.604412 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:29.097383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:29.097345 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:29.680739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:29.680696 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:30.491050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:30.491008 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee757d8-b555-433a-a85b-d6d3bd4ca416" path="/var/lib/kubelet/pods/cee757d8-b555-433a-a85b-d6d3bd4ca416/volumes" Oct 30 17:40:31.421118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:31.421077 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:33.174477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:33.174441 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:35.883076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:35.883035 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9"] Oct 30 17:40:35.883943 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:35.883882 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" containerID="cri-o://ef4f971c5f31624d3fcee2077cf6e99c11d13ab3225ca55d4de7398b07d0ff50" gracePeriod=30 Oct 30 17:40:36.444214 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.444172 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768"] Oct 30 17:40:36.444855 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.444836 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerName="istio-proxy" Oct 30 17:40:36.444919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.444858 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerName="istio-proxy" Oct 30 17:40:36.444919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.444878 4828 cpu_manager.go:414] "containerMap: removing container" podUID="cee757d8-b555-433a-a85b-d6d3bd4ca416" containerName="revoke-keys" Oct 30 17:40:36.444919 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.444888 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee757d8-b555-433a-a85b-d6d3bd4ca416" containerName="revoke-keys" Oct 30 17:40:36.445029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.444938 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerName="istio-proxy" Oct 30 17:40:36.445029 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.444949 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerName="istio-proxy" Oct 30 17:40:36.445137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.445079 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c249be1c-0e0f-4380-917c-bb42dc4308c8" containerName="istio-proxy" Oct 30 17:40:36.445137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.445096 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerName="istio-proxy" Oct 30 17:40:36.445137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.445112 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5afa0e3-1eda-4826-bcf7-1225ab2e50f5" containerName="istio-proxy" Oct 30 17:40:36.445137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.445122 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee757d8-b555-433a-a85b-d6d3bd4ca416" containerName="revoke-keys" Oct 30 17:40:36.448804 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.448784 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" Oct 30 17:40:36.465695 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.465657 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:40:36.465862 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.465760 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:40:36.467571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.467542 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768"] Oct 30 17:40:36.494173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.494124 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-727dv\" (UniqueName: \"kubernetes.io/projected/4ee20410-a23f-417d-b53a-ae0fbe01e315-kube-api-access-727dv\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768\" (UID: \"4ee20410-a23f-417d-b53a-ae0fbe01e315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" Oct 30 17:40:36.494396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.494185 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768\" (UID: \"4ee20410-a23f-417d-b53a-ae0fbe01e315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" Oct 30 17:40:36.494396 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.494310 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768\" (UID: \"4ee20410-a23f-417d-b53a-ae0fbe01e315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" Oct 30 17:40:36.595635 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.595589 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768\" (UID: \"4ee20410-a23f-417d-b53a-ae0fbe01e315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" Oct 30 17:40:36.595855 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.595677 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-727dv\" (UniqueName: \"kubernetes.io/projected/4ee20410-a23f-417d-b53a-ae0fbe01e315-kube-api-access-727dv\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768\" (UID: \"4ee20410-a23f-417d-b53a-ae0fbe01e315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" Oct 30 17:40:36.595855 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.595736 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768\" (UID: \"4ee20410-a23f-417d-b53a-ae0fbe01e315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" Oct 30 17:40:36.596359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.596332 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768\" (UID: \"4ee20410-a23f-417d-b53a-ae0fbe01e315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" Oct 30 17:40:36.598327 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.598301 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768\" (UID: \"4ee20410-a23f-417d-b53a-ae0fbe01e315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" Oct 30 17:40:36.619891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.619859 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-727dv\" (UniqueName: \"kubernetes.io/projected/4ee20410-a23f-417d-b53a-ae0fbe01e315-kube-api-access-727dv\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768\" (UID: \"4ee20410-a23f-417d-b53a-ae0fbe01e315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" Oct 30 17:40:36.761794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:36.761762 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" Oct 30 17:40:37.056586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:37.056486 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:37.065095 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:37.065068 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768"] Oct 30 17:40:37.065566 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:40:37.065526 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee20410_a23f_417d_b53a_ae0fbe01e315.slice/crio-a920750a095563c8c7b7d1e70d2050445eb3d338946e9a94feb76f8bc10cbdb0 WatchSource:0}: Error finding container a920750a095563c8c7b7d1e70d2050445eb3d338946e9a94feb76f8bc10cbdb0: Status 404 returned error can't find the container with id a920750a095563c8c7b7d1e70d2050445eb3d338946e9a94feb76f8bc10cbdb0 Oct 30 17:40:37.203590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:37.203559 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:37.245664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:37.245625 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" event={"ID":"4ee20410-a23f-417d-b53a-ae0fbe01e315","Type":"ContainerStarted","Data":"a920750a095563c8c7b7d1e70d2050445eb3d338946e9a94feb76f8bc10cbdb0"} Oct 30 17:40:37.912068 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:40:37.912032 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:40:38.045601 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:38.045560 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:38.252646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:38.252555 4828 generic.go:356] "Generic (PLEG): container finished" podID="4ee20410-a23f-417d-b53a-ae0fbe01e315" containerID="4c2405e900a055bd10c8ff048cc6561e8ac75fa1b79c7b4752c717ac7a58aedb" exitCode=0 Oct 30 17:40:38.253069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:38.252642 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" event={"ID":"4ee20410-a23f-417d-b53a-ae0fbe01e315","Type":"ContainerDied","Data":"4c2405e900a055bd10c8ff048cc6561e8ac75fa1b79c7b4752c717ac7a58aedb"} Oct 30 17:40:39.390785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:39.390758 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" Oct 30 17:40:39.525501 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:39.525421 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-openshift-ca\") pod \"4ee20410-a23f-417d-b53a-ae0fbe01e315\" (UID: \"4ee20410-a23f-417d-b53a-ae0fbe01e315\") " Oct 30 17:40:39.525659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:39.525533 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee20410-a23f-417d-b53a-ae0fbe01e315-kube-api-access-727dv\" (UniqueName: \"kubernetes.io/projected/4ee20410-a23f-417d-b53a-ae0fbe01e315-kube-api-access-727dv\") pod \"4ee20410-a23f-417d-b53a-ae0fbe01e315\" (UID: \"4ee20410-a23f-417d-b53a-ae0fbe01e315\") " Oct 30 17:40:39.525659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:39.525588 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-xks-ca\") pod \"4ee20410-a23f-417d-b53a-ae0fbe01e315\" (UID: \"4ee20410-a23f-417d-b53a-ae0fbe01e315\") " Oct 30 17:40:39.525878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:39.525850 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-openshift-ca" pod "4ee20410-a23f-417d-b53a-ae0fbe01e315" (UID: "4ee20410-a23f-417d-b53a-ae0fbe01e315"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:39.527637 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:39.527608 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-xks-ca" pod "4ee20410-a23f-417d-b53a-ae0fbe01e315" (UID: "4ee20410-a23f-417d-b53a-ae0fbe01e315"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:40:39.527756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:39.527662 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee20410-a23f-417d-b53a-ae0fbe01e315-kube-api-access-727dv" pod "4ee20410-a23f-417d-b53a-ae0fbe01e315" (UID: "4ee20410-a23f-417d-b53a-ae0fbe01e315"). InnerVolumeSpecName "kube-api-access-727dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:39.626971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:39.626936 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-727dv\" (UniqueName: \"kubernetes.io/projected/4ee20410-a23f-417d-b53a-ae0fbe01e315-kube-api-access-727dv\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:39.626971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:39.626967 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-xks-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:39.626971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:39.626977 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ee20410-a23f-417d-b53a-ae0fbe01e315-maas-api-openshift-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:39.852742 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:39.852652 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:40.265456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:40.265420 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" event={"ID":"4ee20410-a23f-417d-b53a-ae0fbe01e315","Type":"ContainerDied","Data":"a920750a095563c8c7b7d1e70d2050445eb3d338946e9a94feb76f8bc10cbdb0"} Oct 30 17:40:40.265456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:40.265439 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768" Oct 30 17:40:40.265659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:40.265475 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a920750a095563c8c7b7d1e70d2050445eb3d338946e9a94feb76f8bc10cbdb0" Oct 30 17:40:41.150416 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.150390 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:40:41.240964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.240869 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-socket\") pod \"54e68535-8263-4634-9c81-174743d3de5e\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " Oct 30 17:40:41.240964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.240938 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54e68535-8263-4634-9c81-174743d3de5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54e68535-8263-4634-9c81-174743d3de5e-istio-podinfo\") pod \"54e68535-8263-4634-9c81-174743d3de5e\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " Oct 30 17:40:41.241195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.240981 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-certs\") pod \"54e68535-8263-4634-9c81-174743d3de5e\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " Oct 30 17:40:41.241195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241018 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-data\") pod \"54e68535-8263-4634-9c81-174743d3de5e\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " Oct 30 17:40:41.241195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241052 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-credential-socket\") pod \"54e68535-8263-4634-9c81-174743d3de5e\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " Oct 30 17:40:41.241195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241106 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-kube-api-access-zjszj\" (UniqueName: \"kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-kube-api-access-zjszj\") pod \"54e68535-8263-4634-9c81-174743d3de5e\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " Oct 30 17:40:41.241195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241153 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-envoy\") pod \"54e68535-8263-4634-9c81-174743d3de5e\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " Oct 30 17:40:41.241195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241163 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-socket" pod "54e68535-8263-4634-9c81-174743d3de5e" (UID: "54e68535-8263-4634-9c81-174743d3de5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:41.241533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241316 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e68535-8263-4634-9c81-174743d3de5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54e68535-8263-4634-9c81-174743d3de5e-istiod-ca-cert\") pod \"54e68535-8263-4634-9c81-174743d3de5e\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " Oct 30 17:40:41.241533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241376 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-certs" pod "54e68535-8263-4634-9c81-174743d3de5e" (UID: "54e68535-8263-4634-9c81-174743d3de5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:41.241533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241376 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-credential-socket" pod "54e68535-8263-4634-9c81-174743d3de5e" (UID: "54e68535-8263-4634-9c81-174743d3de5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:41.241533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241381 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-istio-token\" (UniqueName: \"kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-istio-token\") pod \"54e68535-8263-4634-9c81-174743d3de5e\" (UID: \"54e68535-8263-4634-9c81-174743d3de5e\") " Oct 30 17:40:41.241533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241412 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-data" pod "54e68535-8263-4634-9c81-174743d3de5e" (UID: "54e68535-8263-4634-9c81-174743d3de5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:41.241818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241763 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e68535-8263-4634-9c81-174743d3de5e-istiod-ca-cert" pod "54e68535-8263-4634-9c81-174743d3de5e" (UID: "54e68535-8263-4634-9c81-174743d3de5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:40:41.241880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241820 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:41.241880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241841 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:41.241880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241857 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:41.241880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.241872 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:41.243237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.243209 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54e68535-8263-4634-9c81-174743d3de5e-istio-podinfo" pod "54e68535-8263-4634-9c81-174743d3de5e" (UID: "54e68535-8263-4634-9c81-174743d3de5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:40:41.243355 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.243319 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-envoy" pod "54e68535-8263-4634-9c81-174743d3de5e" (UID: "54e68535-8263-4634-9c81-174743d3de5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:40:41.243404 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.243392 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-kube-api-access-zjszj" pod "54e68535-8263-4634-9c81-174743d3de5e" (UID: "54e68535-8263-4634-9c81-174743d3de5e"). InnerVolumeSpecName "kube-api-access-zjszj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:41.243540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.243520 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-istio-token" pod "54e68535-8263-4634-9c81-174743d3de5e" (UID: "54e68535-8263-4634-9c81-174743d3de5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:40:41.272205 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.272176 4828 generic.go:356] "Generic (PLEG): container finished" podID="54e68535-8263-4634-9c81-174743d3de5e" containerID="ef4f971c5f31624d3fcee2077cf6e99c11d13ab3225ca55d4de7398b07d0ff50" exitCode=0 Oct 30 17:40:41.272349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.272243 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" Oct 30 17:40:41.272349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.272243 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" event={"ID":"54e68535-8263-4634-9c81-174743d3de5e","Type":"ContainerDied","Data":"ef4f971c5f31624d3fcee2077cf6e99c11d13ab3225ca55d4de7398b07d0ff50"} Oct 30 17:40:41.272468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.272364 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9" event={"ID":"54e68535-8263-4634-9c81-174743d3de5e","Type":"ContainerDied","Data":"4999070a281190fa776957b02713cac6506d71bd5d209c0060db89fb1e8f9ed0"} Oct 30 17:40:41.272468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.272387 4828 scope.go:122] "RemoveContainer" containerID="ef4f971c5f31624d3fcee2077cf6e99c11d13ab3225ca55d4de7398b07d0ff50" Oct 30 17:40:41.282140 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.282120 4828 scope.go:122] "RemoveContainer" containerID="f08005d96cef9ec2453eaa4d9fc5db754dc1e35cde3d67e4fc33e6adf487f8bb" Oct 30 17:40:41.290751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.290733 4828 scope.go:122] "RemoveContainer" containerID="ef4f971c5f31624d3fcee2077cf6e99c11d13ab3225ca55d4de7398b07d0ff50" Oct 30 17:40:41.291079 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:40:41.291057 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef4f971c5f31624d3fcee2077cf6e99c11d13ab3225ca55d4de7398b07d0ff50\": container with ID starting with ef4f971c5f31624d3fcee2077cf6e99c11d13ab3225ca55d4de7398b07d0ff50 not found: ID does not exist" containerID="ef4f971c5f31624d3fcee2077cf6e99c11d13ab3225ca55d4de7398b07d0ff50" Oct 30 17:40:41.291161 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.291087 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef4f971c5f31624d3fcee2077cf6e99c11d13ab3225ca55d4de7398b07d0ff50"} err="failed to get container status \"ef4f971c5f31624d3fcee2077cf6e99c11d13ab3225ca55d4de7398b07d0ff50\": rpc error: code = NotFound desc = could not find container \"ef4f971c5f31624d3fcee2077cf6e99c11d13ab3225ca55d4de7398b07d0ff50\": container with ID starting with ef4f971c5f31624d3fcee2077cf6e99c11d13ab3225ca55d4de7398b07d0ff50 not found: ID does not exist" Oct 30 17:40:41.291161 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.291105 4828 scope.go:122] "RemoveContainer" containerID="f08005d96cef9ec2453eaa4d9fc5db754dc1e35cde3d67e4fc33e6adf487f8bb" Oct 30 17:40:41.291370 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:40:41.291351 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08005d96cef9ec2453eaa4d9fc5db754dc1e35cde3d67e4fc33e6adf487f8bb\": container with ID starting with f08005d96cef9ec2453eaa4d9fc5db754dc1e35cde3d67e4fc33e6adf487f8bb not found: ID does not exist" containerID="f08005d96cef9ec2453eaa4d9fc5db754dc1e35cde3d67e4fc33e6adf487f8bb" Oct 30 17:40:41.291429 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.291380 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08005d96cef9ec2453eaa4d9fc5db754dc1e35cde3d67e4fc33e6adf487f8bb"} err="failed to get container status \"f08005d96cef9ec2453eaa4d9fc5db754dc1e35cde3d67e4fc33e6adf487f8bb\": rpc error: code = NotFound desc = could not find container \"f08005d96cef9ec2453eaa4d9fc5db754dc1e35cde3d67e4fc33e6adf487f8bb\": container with ID starting with f08005d96cef9ec2453eaa4d9fc5db754dc1e35cde3d67e4fc33e6adf487f8bb not found: ID does not exist" Oct 30 17:40:41.330209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.330180 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9"] Oct 30 17:40:41.336005 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.335976 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b87f3c-openshift-default-67f845964f-96gf9"] Oct 30 17:40:41.342407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.342384 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54e68535-8263-4634-9c81-174743d3de5e-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:41.342407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.342408 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54e68535-8263-4634-9c81-174743d3de5e-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:41.342550 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.342417 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:41.342550 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.342425 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54e68535-8263-4634-9c81-174743d3de5e-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:41.342550 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.342434 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjszj\" (UniqueName: \"kubernetes.io/projected/54e68535-8263-4634-9c81-174743d3de5e-kube-api-access-zjszj\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:40:41.500271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:41.500163 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:42.489374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:42.489340 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e68535-8263-4634-9c81-174743d3de5e" path="/var/lib/kubelet/pods/54e68535-8263-4634-9c81-174743d3de5e/volumes" Oct 30 17:40:43.243449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:43.243410 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:44.782770 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:44.782733 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:46.313077 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:46.313030 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:46.721583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:46.721495 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:46.876847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:46.876803 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:47.770257 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:47.770215 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:48.276008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:48.275962 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:48.374169 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:48.374132 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768"] Oct 30 17:40:48.382806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:48.382772 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e13a51-rs768"] Oct 30 17:40:48.432977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:48.432942 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:48.491085 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:48.491051 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee20410-a23f-417d-b53a-ae0fbe01e315" path="/var/lib/kubelet/pods/4ee20410-a23f-417d-b53a-ae0fbe01e315/volumes" Oct 30 17:40:48.596879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:48.596788 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:48.936081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:48.935988 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:50.248851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:50.248813 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:51.932236 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:51.932186 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:53.478861 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:53.478820 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:55.059549 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:55.059505 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:55.895265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:55.895222 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:56.045054 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:56.045017 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:56.278798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:56.278760 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:56.773687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:56.773649 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:56.942582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:56.942537 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:40:58.589122 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:40:58.589083 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:00.344471 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:00.344431 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:00.773990 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:00.773953 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:01.418487 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:01.418453 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:01.849043 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:01.849004 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:01.999797 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:01.999765 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:03.573794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:03.573743 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:05.114112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:05.114061 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:06.682363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:06.682280 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:07.062570 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:07.062524 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h"] Oct 30 17:41:07.211642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:07.211608 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:07.372008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:07.371891 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:07.910258 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:41:07.910222 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:41:08.300248 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:08.300212 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:09.678004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.677956 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq"] Oct 30 17:41:09.678476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.678461 4828 cpu_manager.go:414] "containerMap: removing container" podUID="4ee20410-a23f-417d-b53a-ae0fbe01e315" containerName="revoke-keys" Oct 30 17:41:09.678523 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.678477 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee20410-a23f-417d-b53a-ae0fbe01e315" containerName="revoke-keys" Oct 30 17:41:09.678523 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.678518 4828 cpu_manager.go:414] "containerMap: removing container" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" Oct 30 17:41:09.678601 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.678523 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" Oct 30 17:41:09.678601 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.678535 4828 cpu_manager.go:414] "containerMap: removing container" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" Oct 30 17:41:09.678601 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.678541 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" Oct 30 17:41:09.678702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.678606 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" Oct 30 17:41:09.678702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.678613 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e68535-8263-4634-9c81-174743d3de5e" containerName="istio-proxy" Oct 30 17:41:09.678702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.678622 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3660fa4-e79d-4624-a792-d60275ef4c43" containerName="istio-proxy" Oct 30 17:41:09.678702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.678629 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee20410-a23f-417d-b53a-ae0fbe01e315" containerName="revoke-keys" Oct 30 17:41:09.681784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.681763 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" Oct 30 17:41:09.697407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.697377 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:41:09.697524 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.697412 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kw4fd\"" Oct 30 17:41:09.712657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.712625 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq"] Oct 30 17:41:09.731518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.731483 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-627571c3-2l4nq\" (UID: \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" Oct 30 17:41:09.731697 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.731547 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsjwc\" (UniqueName: \"kubernetes.io/projected/5322d132-73a0-45e0-a7f8-0a82949e8ff2-kube-api-access-nsjwc\") pod \"maas-api-revoke-sub-627571c3-2l4nq\" (UID: \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" Oct 30 17:41:09.731697 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.731652 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-627571c3-2l4nq\" (UID: \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" Oct 30 17:41:09.832657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.832617 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsjwc\" (UniqueName: \"kubernetes.io/projected/5322d132-73a0-45e0-a7f8-0a82949e8ff2-kube-api-access-nsjwc\") pod \"maas-api-revoke-sub-627571c3-2l4nq\" (UID: \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" Oct 30 17:41:09.832870 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.832698 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-627571c3-2l4nq\" (UID: \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" Oct 30 17:41:09.832870 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.832762 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-627571c3-2l4nq\" (UID: \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" Oct 30 17:41:09.833424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.833382 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-627571c3-2l4nq\" (UID: \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" Oct 30 17:41:09.835279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.835255 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-627571c3-2l4nq\" (UID: \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" Oct 30 17:41:09.857430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.857405 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsjwc\" (UniqueName: \"kubernetes.io/projected/5322d132-73a0-45e0-a7f8-0a82949e8ff2-kube-api-access-nsjwc\") pod \"maas-api-revoke-sub-627571c3-2l4nq\" (UID: \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" Oct 30 17:41:09.997755 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:09.997653 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" Oct 30 17:41:10.304630 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:10.304587 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq"] Oct 30 17:41:10.305143 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:41:10.305117 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5322d132_73a0_45e0_a7f8_0a82949e8ff2.slice/crio-1261525ef5e109a001f01dd69bdda9c4d6ee7175afacf03ddd48a00fe8532bda WatchSource:0}: Error finding container 1261525ef5e109a001f01dd69bdda9c4d6ee7175afacf03ddd48a00fe8532bda: Status 404 returned error can't find the container with id 1261525ef5e109a001f01dd69bdda9c4d6ee7175afacf03ddd48a00fe8532bda Oct 30 17:41:10.437320 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:10.437282 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" event={"ID":"5322d132-73a0-45e0-a7f8-0a82949e8ff2","Type":"ContainerStarted","Data":"1261525ef5e109a001f01dd69bdda9c4d6ee7175afacf03ddd48a00fe8532bda"} Oct 30 17:41:11.446984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:11.446880 4828 generic.go:356] "Generic (PLEG): container finished" podID="5322d132-73a0-45e0-a7f8-0a82949e8ff2" containerID="47c697274b60990be99d3e1090e8a66eeaa488383ccd7b4f2c420c90694d24d3" exitCode=0 Oct 30 17:41:11.446984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:11.446938 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" event={"ID":"5322d132-73a0-45e0-a7f8-0a82949e8ff2","Type":"ContainerDied","Data":"47c697274b60990be99d3e1090e8a66eeaa488383ccd7b4f2c420c90694d24d3"} Oct 30 17:41:12.063746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:12.063704 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" podUID="6eb2dd00-1748-4fee-a0c5-1a600bed0c18" containerName="payload-pre-processing" containerID="cri-o://7bd635e7b6ca3cf218a3ab2284bf9cd718a46404fdd028269a49052dc72362eb" gracePeriod=25 Oct 30 17:41:12.601705 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:12.601678 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" Oct 30 17:41:12.663895 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:12.663862 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5322d132-73a0-45e0-a7f8-0a82949e8ff2-kube-api-access-nsjwc\" (UniqueName: \"kubernetes.io/projected/5322d132-73a0-45e0-a7f8-0a82949e8ff2-kube-api-access-nsjwc\") pod \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\" (UID: \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\") " Oct 30 17:41:12.664093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:12.664008 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-openshift-ca\") pod \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\" (UID: \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\") " Oct 30 17:41:12.664093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:12.664080 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-xks-ca\") pod \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\" (UID: \"5322d132-73a0-45e0-a7f8-0a82949e8ff2\") " Oct 30 17:41:12.664430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:12.664401 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-openshift-ca" pod "5322d132-73a0-45e0-a7f8-0a82949e8ff2" (UID: "5322d132-73a0-45e0-a7f8-0a82949e8ff2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:12.666232 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:12.666209 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-xks-ca" pod "5322d132-73a0-45e0-a7f8-0a82949e8ff2" (UID: "5322d132-73a0-45e0-a7f8-0a82949e8ff2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:41:12.666332 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:12.666266 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5322d132-73a0-45e0-a7f8-0a82949e8ff2-kube-api-access-nsjwc" pod "5322d132-73a0-45e0-a7f8-0a82949e8ff2" (UID: "5322d132-73a0-45e0-a7f8-0a82949e8ff2"). InnerVolumeSpecName "kube-api-access-nsjwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:12.765381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:12.765350 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-openshift-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:12.765381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:12.765379 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5322d132-73a0-45e0-a7f8-0a82949e8ff2-maas-api-xks-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:12.765590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:12.765389 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsjwc\" (UniqueName: \"kubernetes.io/projected/5322d132-73a0-45e0-a7f8-0a82949e8ff2-kube-api-access-nsjwc\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:13.459298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:13.459215 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" Oct 30 17:41:13.459298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:13.459251 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq" event={"ID":"5322d132-73a0-45e0-a7f8-0a82949e8ff2","Type":"ContainerDied","Data":"1261525ef5e109a001f01dd69bdda9c4d6ee7175afacf03ddd48a00fe8532bda"} Oct 30 17:41:13.459493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:13.459333 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1261525ef5e109a001f01dd69bdda9c4d6ee7175afacf03ddd48a00fe8532bda" Oct 30 17:41:14.240345 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:14.240302 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" podUID="6eb2dd00-1748-4fee-a0c5-1a600bed0c18" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:41:23.630493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:23.630450 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:23.788246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:23.788189 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:24.240526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:24.240485 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" podUID="6eb2dd00-1748-4fee-a0c5-1a600bed0c18" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:41:24.955403 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:24.955357 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:25.409324 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:25.409275 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:25.561088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:25.561031 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:26.047053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:26.047018 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:32.323878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.323798 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" Oct 30 17:41:32.459546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.459510 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-plugins-config-volume\") pod \"6eb2dd00-1748-4fee-a0c5-1a600bed0c18\" (UID: \"6eb2dd00-1748-4fee-a0c5-1a600bed0c18\") " Oct 30 17:41:32.459732 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.459560 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-kube-api-access-g8fmw\" (UniqueName: \"kubernetes.io/projected/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-kube-api-access-g8fmw\") pod \"6eb2dd00-1748-4fee-a0c5-1a600bed0c18\" (UID: \"6eb2dd00-1748-4fee-a0c5-1a600bed0c18\") " Oct 30 17:41:32.459963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.459936 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-plugins-config-volume" pod "6eb2dd00-1748-4fee-a0c5-1a600bed0c18" (UID: "6eb2dd00-1748-4fee-a0c5-1a600bed0c18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:32.461824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.461798 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-kube-api-access-g8fmw" pod "6eb2dd00-1748-4fee-a0c5-1a600bed0c18" (UID: "6eb2dd00-1748-4fee-a0c5-1a600bed0c18"). InnerVolumeSpecName "kube-api-access-g8fmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:32.560499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.560450 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:32.560499 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.560484 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8fmw\" (UniqueName: \"kubernetes.io/projected/6eb2dd00-1748-4fee-a0c5-1a600bed0c18-kube-api-access-g8fmw\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:32.566424 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.566393 4828 generic.go:356] "Generic (PLEG): container finished" podID="6eb2dd00-1748-4fee-a0c5-1a600bed0c18" containerID="7bd635e7b6ca3cf218a3ab2284bf9cd718a46404fdd028269a49052dc72362eb" exitCode=0 Oct 30 17:41:32.566645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.566443 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" Oct 30 17:41:32.566645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.566467 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" event={"ID":"6eb2dd00-1748-4fee-a0c5-1a600bed0c18","Type":"ContainerDied","Data":"7bd635e7b6ca3cf218a3ab2284bf9cd718a46404fdd028269a49052dc72362eb"} Oct 30 17:41:32.566645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.566519 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h" event={"ID":"6eb2dd00-1748-4fee-a0c5-1a600bed0c18","Type":"ContainerDied","Data":"4ef77522db7db95f57fd0ee7ab2211142515bdd529eeb0e4252fe72d8b1c8a01"} Oct 30 17:41:32.566645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.566540 4828 scope.go:122] "RemoveContainer" containerID="7bd635e7b6ca3cf218a3ab2284bf9cd718a46404fdd028269a49052dc72362eb" Oct 30 17:41:32.575984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.575960 4828 scope.go:122] "RemoveContainer" containerID="7bd635e7b6ca3cf218a3ab2284bf9cd718a46404fdd028269a49052dc72362eb" Oct 30 17:41:32.576297 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:41:32.576273 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd635e7b6ca3cf218a3ab2284bf9cd718a46404fdd028269a49052dc72362eb\": container with ID starting with 7bd635e7b6ca3cf218a3ab2284bf9cd718a46404fdd028269a49052dc72362eb not found: ID does not exist" containerID="7bd635e7b6ca3cf218a3ab2284bf9cd718a46404fdd028269a49052dc72362eb" Oct 30 17:41:32.576345 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.576308 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd635e7b6ca3cf218a3ab2284bf9cd718a46404fdd028269a49052dc72362eb"} err="failed to get container status \"7bd635e7b6ca3cf218a3ab2284bf9cd718a46404fdd028269a49052dc72362eb\": rpc error: code = NotFound desc = could not find container \"7bd635e7b6ca3cf218a3ab2284bf9cd718a46404fdd028269a49052dc72362eb\": container with ID starting with 7bd635e7b6ca3cf218a3ab2284bf9cd718a46404fdd028269a49052dc72362eb not found: ID does not exist" Oct 30 17:41:32.621299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.621261 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h"] Oct 30 17:41:32.625784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:32.625736 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-d384dc-5dc59899c5-c2z9h"] Oct 30 17:41:33.036834 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:33.036795 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:33.191772 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:33.191733 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:33.340067 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:33.339952 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw"] Oct 30 17:41:33.340632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:33.340244 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" podUID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" containerName="istio-proxy" containerID="cri-o://e0f7bc0e13a76fb6e65db58ce02b70b07cf030b7771946f2c57c1f758a40f260" gracePeriod=30 Oct 30 17:41:33.449796 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:33.449761 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:34.489734 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:34.489693 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb2dd00-1748-4fee-a0c5-1a600bed0c18" path="/var/lib/kubelet/pods/6eb2dd00-1748-4fee-a0c5-1a600bed0c18/volumes" Oct 30 17:41:36.089860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.089821 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78487d6d64-hdjk9"] Oct 30 17:41:36.090323 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.090035 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-78487d6d64-hdjk9" podUID="41a0571b-6e0f-4fa9-bbd3-d0b31614b19b" containerName="manager" containerID="cri-o://79a08b1bd537066ad655e5be4faa997c7746d8b67baa818b8e044ae0c0f50b51" gracePeriod=30 Oct 30 17:41:36.355615 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.355553 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 17:41:36.501068 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.501036 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-webhook-cert\") pod \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\" (UID: \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\") " Oct 30 17:41:36.501263 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.501092 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-metrics-certs\") pod \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\" (UID: \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\") " Oct 30 17:41:36.501263 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.501118 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-kube-api-access-kd7b7\" (UniqueName: \"kubernetes.io/projected/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-kube-api-access-kd7b7\") pod \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\" (UID: \"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b\") " Oct 30 17:41:36.503195 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.503162 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-webhook-cert" pod "41a0571b-6e0f-4fa9-bbd3-d0b31614b19b" (UID: "41a0571b-6e0f-4fa9-bbd3-d0b31614b19b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:41:36.503318 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.503231 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-kube-api-access-kd7b7" pod "41a0571b-6e0f-4fa9-bbd3-d0b31614b19b" (UID: "41a0571b-6e0f-4fa9-bbd3-d0b31614b19b"). InnerVolumeSpecName "kube-api-access-kd7b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:36.503318 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.503253 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-metrics-certs" pod "41a0571b-6e0f-4fa9-bbd3-d0b31614b19b" (UID: "41a0571b-6e0f-4fa9-bbd3-d0b31614b19b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:41:36.595853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.595813 4828 generic.go:356] "Generic (PLEG): container finished" podID="41a0571b-6e0f-4fa9-bbd3-d0b31614b19b" containerID="79a08b1bd537066ad655e5be4faa997c7746d8b67baa818b8e044ae0c0f50b51" exitCode=0 Oct 30 17:41:36.596070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.595858 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78487d6d64-hdjk9" Oct 30 17:41:36.596070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.595917 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78487d6d64-hdjk9" event={"ID":"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b","Type":"ContainerDied","Data":"79a08b1bd537066ad655e5be4faa997c7746d8b67baa818b8e044ae0c0f50b51"} Oct 30 17:41:36.596070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.595975 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78487d6d64-hdjk9" event={"ID":"41a0571b-6e0f-4fa9-bbd3-d0b31614b19b","Type":"ContainerDied","Data":"dbf78ee80e0f4773ef61112c2c0b8d7f8b2658ad1c7f580c11a0f86828240d7c"} Oct 30 17:41:36.596070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.595993 4828 scope.go:122] "RemoveContainer" containerID="79a08b1bd537066ad655e5be4faa997c7746d8b67baa818b8e044ae0c0f50b51" Oct 30 17:41:36.602102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.602075 4828 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-webhook-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:36.602102 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.602103 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:36.602331 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.602190 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd7b7\" (UniqueName: \"kubernetes.io/projected/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b-kube-api-access-kd7b7\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:36.607469 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.607446 4828 scope.go:122] "RemoveContainer" containerID="79a08b1bd537066ad655e5be4faa997c7746d8b67baa818b8e044ae0c0f50b51" Oct 30 17:41:36.607744 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:41:36.607717 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a08b1bd537066ad655e5be4faa997c7746d8b67baa818b8e044ae0c0f50b51\": container with ID starting with 79a08b1bd537066ad655e5be4faa997c7746d8b67baa818b8e044ae0c0f50b51 not found: ID does not exist" containerID="79a08b1bd537066ad655e5be4faa997c7746d8b67baa818b8e044ae0c0f50b51" Oct 30 17:41:36.607842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.607750 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a08b1bd537066ad655e5be4faa997c7746d8b67baa818b8e044ae0c0f50b51"} err="failed to get container status \"79a08b1bd537066ad655e5be4faa997c7746d8b67baa818b8e044ae0c0f50b51\": rpc error: code = NotFound desc = could not find container \"79a08b1bd537066ad655e5be4faa997c7746d8b67baa818b8e044ae0c0f50b51\": container with ID starting with 79a08b1bd537066ad655e5be4faa997c7746d8b67baa818b8e044ae0c0f50b51 not found: ID does not exist" Oct 30 17:41:36.649811 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.649776 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78487d6d64-hdjk9"] Oct 30 17:41:36.654475 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:36.654439 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78487d6d64-hdjk9"] Oct 30 17:41:37.311380 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.311337 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78487d6d64-fkx6f"] Oct 30 17:41:37.311982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.311961 4828 cpu_manager.go:414] "containerMap: removing container" podUID="5322d132-73a0-45e0-a7f8-0a82949e8ff2" containerName="revoke-keys" Oct 30 17:41:37.312105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.311984 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5322d132-73a0-45e0-a7f8-0a82949e8ff2" containerName="revoke-keys" Oct 30 17:41:37.312105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.312025 4828 cpu_manager.go:414] "containerMap: removing container" podUID="6eb2dd00-1748-4fee-a0c5-1a600bed0c18" containerName="payload-pre-processing" Oct 30 17:41:37.312105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.312033 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb2dd00-1748-4fee-a0c5-1a600bed0c18" containerName="payload-pre-processing" Oct 30 17:41:37.312105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.312047 4828 cpu_manager.go:414] "containerMap: removing container" podUID="41a0571b-6e0f-4fa9-bbd3-d0b31614b19b" containerName="manager" Oct 30 17:41:37.312105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.312054 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a0571b-6e0f-4fa9-bbd3-d0b31614b19b" containerName="manager" Oct 30 17:41:37.312374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.312177 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="5322d132-73a0-45e0-a7f8-0a82949e8ff2" containerName="revoke-keys" Oct 30 17:41:37.312374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.312197 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb2dd00-1748-4fee-a0c5-1a600bed0c18" containerName="payload-pre-processing" Oct 30 17:41:37.312374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.312208 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a0571b-6e0f-4fa9-bbd3-d0b31614b19b" containerName="manager" Oct 30 17:41:37.325760 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.325718 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:41:37.345890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.345856 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78487d6d64-fkx6f"] Oct 30 17:41:37.354798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.354768 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 17:41:37.354997 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.354970 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8xm5f\"" Oct 30 17:41:37.355087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.354994 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 17:41:37.355087 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.355004 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 17:41:37.409841 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.409797 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-metrics-certs\") pod \"maas-controller-78487d6d64-fkx6f\" (UID: \"ec0507da-9433-43b6-9f30-be94f7e1cf06\") " pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:41:37.410007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.409858 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-webhook-cert\") pod \"maas-controller-78487d6d64-fkx6f\" (UID: \"ec0507da-9433-43b6-9f30-be94f7e1cf06\") " pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:41:37.410076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.410004 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks74g\" (UniqueName: \"kubernetes.io/projected/ec0507da-9433-43b6-9f30-be94f7e1cf06-kube-api-access-ks74g\") pod \"maas-controller-78487d6d64-fkx6f\" (UID: \"ec0507da-9433-43b6-9f30-be94f7e1cf06\") " pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:41:37.511173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.511136 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-metrics-certs\") pod \"maas-controller-78487d6d64-fkx6f\" (UID: \"ec0507da-9433-43b6-9f30-be94f7e1cf06\") " pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:41:37.511173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.511185 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-webhook-cert\") pod \"maas-controller-78487d6d64-fkx6f\" (UID: \"ec0507da-9433-43b6-9f30-be94f7e1cf06\") " pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:41:37.511445 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.511219 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks74g\" (UniqueName: \"kubernetes.io/projected/ec0507da-9433-43b6-9f30-be94f7e1cf06-kube-api-access-ks74g\") pod \"maas-controller-78487d6d64-fkx6f\" (UID: \"ec0507da-9433-43b6-9f30-be94f7e1cf06\") " pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:41:37.514036 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.514000 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-webhook-cert\") pod \"maas-controller-78487d6d64-fkx6f\" (UID: \"ec0507da-9433-43b6-9f30-be94f7e1cf06\") " pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:41:37.514177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.514000 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-metrics-certs\") pod \"maas-controller-78487d6d64-fkx6f\" (UID: \"ec0507da-9433-43b6-9f30-be94f7e1cf06\") " pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:41:37.535441 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.535414 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks74g\" (UniqueName: \"kubernetes.io/projected/ec0507da-9433-43b6-9f30-be94f7e1cf06-kube-api-access-ks74g\") pod \"maas-controller-78487d6d64-fkx6f\" (UID: \"ec0507da-9433-43b6-9f30-be94f7e1cf06\") " pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:41:37.644561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.644468 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:41:37.915660 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:41:37.915567 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:41:37.952311 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.951538 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78487d6d64-fkx6f"] Oct 30 17:41:37.953544 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.953510 4828 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:41:37.953780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:37.953688 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:38.140194 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.140132 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:38.490277 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.490247 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a0571b-6e0f-4fa9-bbd3-d0b31614b19b" path="/var/lib/kubelet/pods/41a0571b-6e0f-4fa9-bbd3-d0b31614b19b/volumes" Oct 30 17:41:38.610789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.610759 4828 generic.go:356] "Generic (PLEG): container finished" podID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" containerID="e0f7bc0e13a76fb6e65db58ce02b70b07cf030b7771946f2c57c1f758a40f260" exitCode=0 Oct 30 17:41:38.616075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.610842 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" event={"ID":"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167","Type":"ContainerDied","Data":"e0f7bc0e13a76fb6e65db58ce02b70b07cf030b7771946f2c57c1f758a40f260"} Oct 30 17:41:38.616075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.610938 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" event={"ID":"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167","Type":"ContainerDied","Data":"e83a4372c5d16aa37e7d7ba40878f50afb28eb27f392d4080b030c668912ef68"} Oct 30 17:41:38.616075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.610968 4828 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e83a4372c5d16aa37e7d7ba40878f50afb28eb27f392d4080b030c668912ef68" Oct 30 17:41:38.616075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.612933 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78487d6d64-fkx6f" event={"ID":"ec0507da-9433-43b6-9f30-be94f7e1cf06","Type":"ContainerStarted","Data":"6c7bf74a3948626fa0dbdcda97e0abd8cc3163c9fee437022c4cad400cff1046"} Oct 30 17:41:38.616075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.612978 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78487d6d64-fkx6f" event={"ID":"ec0507da-9433-43b6-9f30-be94f7e1cf06","Type":"ContainerStarted","Data":"2b2a7f52c5108f87aad460759c44a52145dd5a3d9c035f75ed6e504f67a21b46"} Oct 30 17:41:38.616075 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.613177 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:41:38.636817 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.636788 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:41:38.669097 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.667684 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78487d6d64-fkx6f" podStartSLOduration=1.15716239 podStartE2EDuration="1.667662502s" totalImagesPullingTime="510.500112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:38.66571749 +0000 UTC m=+3276.311227672" watchObservedRunningTime="2026-10-30 17:41:38.667662502 +0000 UTC m=+3276.313172682" Oct 30 17:41:38.723619 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.723586 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-kube-api-access-hfp87\" (UniqueName: \"kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-kube-api-access-hfp87\") pod \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " Oct 30 17:41:38.723619 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.723626 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-socket\") pod \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " Oct 30 17:41:38.724026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.723660 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-envoy\") pod \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " Oct 30 17:41:38.724026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.723697 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-data\") pod \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " Oct 30 17:41:38.724026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.723779 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-token\" (UniqueName: \"kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-token\") pod \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " Oct 30 17:41:38.724026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.723918 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-credential-socket\") pod \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " Oct 30 17:41:38.724026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.723965 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istiod-ca-cert\") pod \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " Oct 30 17:41:38.724026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.724018 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-certs\") pod \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " Oct 30 17:41:38.725494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.724054 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-podinfo\") pod \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\" (UID: \"ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167\") " Oct 30 17:41:38.725494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.725159 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-credential-socket" pod "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" (UID: "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:38.725494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.725362 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-data" pod "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" (UID: "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:38.725941 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.725886 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-socket" pod "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" (UID: "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:38.726891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.726860 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-certs" pod "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" (UID: "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:38.727431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.727410 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istiod-ca-cert" pod "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" (UID: "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:41:38.728359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.728332 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-token" pod "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" (UID: "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:38.729536 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.729515 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-podinfo" pod "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" (UID: "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:41:38.731890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.731867 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-kube-api-access-hfp87" pod "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" (UID: "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167"). InnerVolumeSpecName "kube-api-access-hfp87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:41:38.733238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.733212 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-envoy" pod "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" (UID: "ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:41:38.825699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.825662 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:38.825699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.825694 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfp87\" (UniqueName: \"kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-kube-api-access-hfp87\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:38.825699 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.825704 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:38.825961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.825717 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:38.825961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.825737 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:38.825961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.825747 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:38.825961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.825755 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:38.825961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.825762 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:38.825961 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:38.825771 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:41:39.621240 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:39.621191 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw" Oct 30 17:41:39.717319 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:39.717284 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw"] Oct 30 17:41:39.729512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:39.729471 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d384dc-openshift-default-6fb47fb5d9-95dkw"] Oct 30 17:41:40.488273 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:40.488239 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" path="/var/lib/kubelet/pods/ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167/volumes" Oct 30 17:41:40.551508 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:40.551469 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:46.382976 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:46.382934 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:41:49.624019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:49.623984 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:41:54.573093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.573056 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99"] Oct 30 17:41:54.573503 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.573494 4828 cpu_manager.go:414] "containerMap: removing container" podUID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" containerName="istio-proxy" Oct 30 17:41:54.573584 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.573505 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" containerName="istio-proxy" Oct 30 17:41:54.573584 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.573580 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff93c7e0-92d5-4d7a-ba3b-ab36a0ef4167" containerName="istio-proxy" Oct 30 17:41:54.578180 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.578156 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.608392 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.608349 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99"] Oct 30 17:41:54.664782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.664740 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.664782 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.664786 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa4964f-39da-49c6-bcf5-515c41e85e72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.665079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.664849 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.665079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.664926 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.665079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.664977 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgmjl\" (UniqueName: \"kubernetes.io/projected/bfa4964f-39da-49c6-bcf5-515c41e85e72-kube-api-access-tgmjl\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.665079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.665036 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa4964f-39da-49c6-bcf5-515c41e85e72-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.665079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.665078 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.665339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.665098 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa4964f-39da-49c6-bcf5-515c41e85e72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.665339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.665137 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.766081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.766038 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa4964f-39da-49c6-bcf5-515c41e85e72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.766081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.766086 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.766381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.766107 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.766381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.766143 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgmjl\" (UniqueName: \"kubernetes.io/projected/bfa4964f-39da-49c6-bcf5-515c41e85e72-kube-api-access-tgmjl\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.766381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.766165 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa4964f-39da-49c6-bcf5-515c41e85e72-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.766381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.766212 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.766381 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.766333 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa4964f-39da-49c6-bcf5-515c41e85e72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.766671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.766386 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.766671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.766468 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.766671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.766574 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.766866 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.766843 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.766974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.766891 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.767045 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.766990 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.767132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.767109 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa4964f-39da-49c6-bcf5-515c41e85e72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.768562 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.768541 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa4964f-39da-49c6-bcf5-515c41e85e72-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.768723 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.768666 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa4964f-39da-49c6-bcf5-515c41e85e72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.791120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.791083 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa4964f-39da-49c6-bcf5-515c41e85e72-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.791363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.791216 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgmjl\" (UniqueName: \"kubernetes.io/projected/bfa4964f-39da-49c6-bcf5-515c41e85e72-kube-api-access-tgmjl\") pod \"maas-default-gateway-openshift-default-6db586dd6c-nmv99\" (UID: \"bfa4964f-39da-49c6-bcf5-515c41e85e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:54.891527 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:54.891436 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:55.211388 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:55.211354 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99"] Oct 30 17:41:55.212858 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:41:55.212828 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa4964f_39da_49c6_bcf5_515c41e85e72.slice/crio-479133c848a53787377eec4c2afc252f0c895d6c7f4b875840bea86c88aee06d WatchSource:0}: Error finding container 479133c848a53787377eec4c2afc252f0c895d6c7f4b875840bea86c88aee06d: Status 404 returned error can't find the container with id 479133c848a53787377eec4c2afc252f0c895d6c7f4b875840bea86c88aee06d Oct 30 17:41:55.215011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:55.214981 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:41:55.215088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:55.215041 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:41:55.215088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:55.215074 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:41:55.708977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:55.708937 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" event={"ID":"bfa4964f-39da-49c6-bcf5-515c41e85e72","Type":"ContainerStarted","Data":"8cc2727f528a2ee517994a10978c279c1582fff82592efb9bc9e5663ba15d03a"} Oct 30 17:41:55.709389 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:55.708996 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" event={"ID":"bfa4964f-39da-49c6-bcf5-515c41e85e72","Type":"ContainerStarted","Data":"479133c848a53787377eec4c2afc252f0c895d6c7f4b875840bea86c88aee06d"} Oct 30 17:41:55.756518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:55.756466 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" podStartSLOduration=1.756451841 podStartE2EDuration="1.756451841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:41:55.755863072 +0000 UTC m=+3293.401373253" watchObservedRunningTime="2026-10-30 17:41:55.756451841 +0000 UTC m=+3293.401962018" Oct 30 17:41:55.894037 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:55.893994 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:41:56.896088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:56.896047 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nmv99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 17:41:56.896583 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:56.896131 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" podUID="bfa4964f-39da-49c6-bcf5-515c41e85e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 17:41:57.896925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:57.895347 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nmv99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 17:41:57.896925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:57.895432 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" podUID="bfa4964f-39da-49c6-bcf5-515c41e85e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 17:41:58.892924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:58.892331 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nmv99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 17:41:58.892924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:58.892396 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" podUID="bfa4964f-39da-49c6-bcf5-515c41e85e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 17:41:59.894602 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:59.894562 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nmv99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 17:41:59.895245 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:41:59.895212 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" podUID="bfa4964f-39da-49c6-bcf5-515c41e85e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 17:42:00.892926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:00.892307 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nmv99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 17:42:00.892926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:00.892390 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" podUID="bfa4964f-39da-49c6-bcf5-515c41e85e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 17:42:00.940494 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:00.940451 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:42:01.550843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:01.550798 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:42:01.892405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:01.892308 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nmv99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 30 17:42:01.892405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:01.892378 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" podUID="bfa4964f-39da-49c6-bcf5-515c41e85e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 30 17:42:03.893126 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:03.893077 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nmv99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:42:03.893616 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:03.893152 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" podUID="bfa4964f-39da-49c6-bcf5-515c41e85e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:42:04.934184 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:04.934131 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nmv99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:42:04.934722 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:04.934206 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" podUID="bfa4964f-39da-49c6-bcf5-515c41e85e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:42:05.669896 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:05.669862 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:42:05.670321 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:05.670300 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:42:05.671194 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:05.671173 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" Oct 30 17:42:07.911724 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:42:07.911682 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:42:07.979644 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:07.979613 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:42:07.985183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:07.985157 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:42:12.535141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.535097 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:42:12.638209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.638169 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78487d6d64-fkx6f"] Oct 30 17:42:12.638407 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.638383 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-78487d6d64-fkx6f" podUID="ec0507da-9433-43b6-9f30-be94f7e1cf06" containerName="manager" containerID="cri-o://6c7bf74a3948626fa0dbdcda97e0abd8cc3163c9fee437022c4cad400cff1046" gracePeriod=30 Oct 30 17:42:12.810219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.809868 4828 generic.go:356] "Generic (PLEG): container finished" podID="ec0507da-9433-43b6-9f30-be94f7e1cf06" containerID="6c7bf74a3948626fa0dbdcda97e0abd8cc3163c9fee437022c4cad400cff1046" exitCode=0 Oct 30 17:42:12.810219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.809946 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78487d6d64-fkx6f" event={"ID":"ec0507da-9433-43b6-9f30-be94f7e1cf06","Type":"ContainerDied","Data":"6c7bf74a3948626fa0dbdcda97e0abd8cc3163c9fee437022c4cad400cff1046"} Oct 30 17:42:12.897598 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.897566 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:42:12.960340 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.960305 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0507da-9433-43b6-9f30-be94f7e1cf06-kube-api-access-ks74g\" (UniqueName: \"kubernetes.io/projected/ec0507da-9433-43b6-9f30-be94f7e1cf06-kube-api-access-ks74g\") pod \"ec0507da-9433-43b6-9f30-be94f7e1cf06\" (UID: \"ec0507da-9433-43b6-9f30-be94f7e1cf06\") " Oct 30 17:42:12.960542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.960514 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-metrics-certs\") pod \"ec0507da-9433-43b6-9f30-be94f7e1cf06\" (UID: \"ec0507da-9433-43b6-9f30-be94f7e1cf06\") " Oct 30 17:42:12.960629 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.960607 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-webhook-cert\") pod \"ec0507da-9433-43b6-9f30-be94f7e1cf06\" (UID: \"ec0507da-9433-43b6-9f30-be94f7e1cf06\") " Oct 30 17:42:12.962752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.962721 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-webhook-cert" pod "ec0507da-9433-43b6-9f30-be94f7e1cf06" (UID: "ec0507da-9433-43b6-9f30-be94f7e1cf06"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:42:12.962960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.962780 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-metrics-certs" pod "ec0507da-9433-43b6-9f30-be94f7e1cf06" (UID: "ec0507da-9433-43b6-9f30-be94f7e1cf06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:42:12.963124 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.963107 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0507da-9433-43b6-9f30-be94f7e1cf06-kube-api-access-ks74g" pod "ec0507da-9433-43b6-9f30-be94f7e1cf06" (UID: "ec0507da-9433-43b6-9f30-be94f7e1cf06"). InnerVolumeSpecName "kube-api-access-ks74g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:42:12.990231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.990193 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9"] Oct 30 17:42:12.990604 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:12.990551 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" podUID="b32fbfdf-cbf9-4cf3-a261-bce740046772" containerName="discovery" containerID="cri-o://99c3ca732312c8785f696e01bc56fc3827f09f2797e98f65e758b381ef31258d" gracePeriod=30 Oct 30 17:42:13.062306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.062231 4828 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-webhook-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:42:13.062306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.062262 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks74g\" (UniqueName: \"kubernetes.io/projected/ec0507da-9433-43b6-9f30-be94f7e1cf06-kube-api-access-ks74g\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:42:13.062306 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.062271 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec0507da-9433-43b6-9f30-be94f7e1cf06-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:42:13.147359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.147329 4828 scope.go:122] "RemoveContainer" containerID="e0f7bc0e13a76fb6e65db58ce02b70b07cf030b7771946f2c57c1f758a40f260" Oct 30 17:42:13.237873 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.237839 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 17:42:13.365237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.365152 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-token\" (UniqueName: \"kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-token\") pod \"b32fbfdf-cbf9-4cf3-a261-bce740046772\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " Oct 30 17:42:13.365237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.365194 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-dns-cert\") pod \"b32fbfdf-cbf9-4cf3-a261-bce740046772\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " Oct 30 17:42:13.365237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.365233 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-kube-api-access-jh2gp\" (UniqueName: \"kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-kube-api-access-jh2gp\") pod \"b32fbfdf-cbf9-4cf3-a261-bce740046772\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " Oct 30 17:42:13.365534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.365293 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32fbfdf-cbf9-4cf3-a261-bce740046772-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32fbfdf-cbf9-4cf3-a261-bce740046772-local-certs\") pod \"b32fbfdf-cbf9-4cf3-a261-bce740046772\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " Oct 30 17:42:13.365534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.365320 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-cacerts\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-cacerts\") pod \"b32fbfdf-cbf9-4cf3-a261-bce740046772\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " Oct 30 17:42:13.365534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.365380 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-kubeconfig\") pod \"b32fbfdf-cbf9-4cf3-a261-bce740046772\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " Oct 30 17:42:13.365534 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.365453 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-ca-configmap\") pod \"b32fbfdf-cbf9-4cf3-a261-bce740046772\" (UID: \"b32fbfdf-cbf9-4cf3-a261-bce740046772\") " Oct 30 17:42:13.366246 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.366190 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-ca-configmap" pod "b32fbfdf-cbf9-4cf3-a261-bce740046772" (UID: "b32fbfdf-cbf9-4cf3-a261-bce740046772"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:42:13.367671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.367642 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-kube-api-access-jh2gp" pod "b32fbfdf-cbf9-4cf3-a261-bce740046772" (UID: "b32fbfdf-cbf9-4cf3-a261-bce740046772"). InnerVolumeSpecName "kube-api-access-jh2gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:42:13.367801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.367784 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-kubeconfig" pod "b32fbfdf-cbf9-4cf3-a261-bce740046772" (UID: "b32fbfdf-cbf9-4cf3-a261-bce740046772"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:42:13.368023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.367917 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-cacerts" pod "b32fbfdf-cbf9-4cf3-a261-bce740046772" (UID: "b32fbfdf-cbf9-4cf3-a261-bce740046772"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:42:13.368023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.367952 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32fbfdf-cbf9-4cf3-a261-bce740046772-local-certs" pod "b32fbfdf-cbf9-4cf3-a261-bce740046772" (UID: "b32fbfdf-cbf9-4cf3-a261-bce740046772"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:42:13.368023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.367962 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-token" pod "b32fbfdf-cbf9-4cf3-a261-bce740046772" (UID: "b32fbfdf-cbf9-4cf3-a261-bce740046772"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:42:13.368023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.367993 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-dns-cert" pod "b32fbfdf-cbf9-4cf3-a261-bce740046772" (UID: "b32fbfdf-cbf9-4cf3-a261-bce740046772"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:42:13.467267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.467227 4828 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-ca-configmap\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:42:13.467267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.467262 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:42:13.467490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.467276 4828 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-csr-dns-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:42:13.467490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.467288 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh2gp\" (UniqueName: \"kubernetes.io/projected/b32fbfdf-cbf9-4cf3-a261-bce740046772-kube-api-access-jh2gp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:42:13.467490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.467299 4828 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b32fbfdf-cbf9-4cf3-a261-bce740046772-local-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:42:13.467490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.467312 4828 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-cacerts\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:42:13.467490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.467323 4828 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b32fbfdf-cbf9-4cf3-a261-bce740046772-istio-kubeconfig\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:42:13.817031 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.816874 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78487d6d64-fkx6f" Oct 30 17:42:13.817031 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.816943 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78487d6d64-fkx6f" event={"ID":"ec0507da-9433-43b6-9f30-be94f7e1cf06","Type":"ContainerDied","Data":"2b2a7f52c5108f87aad460759c44a52145dd5a3d9c035f75ed6e504f67a21b46"} Oct 30 17:42:13.817031 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.817010 4828 scope.go:122] "RemoveContainer" containerID="6c7bf74a3948626fa0dbdcda97e0abd8cc3163c9fee437022c4cad400cff1046" Oct 30 17:42:13.818413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.818393 4828 generic.go:356] "Generic (PLEG): container finished" podID="b32fbfdf-cbf9-4cf3-a261-bce740046772" containerID="99c3ca732312c8785f696e01bc56fc3827f09f2797e98f65e758b381ef31258d" exitCode=0 Oct 30 17:42:13.818514 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.818443 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" Oct 30 17:42:13.818514 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.818453 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" event={"ID":"b32fbfdf-cbf9-4cf3-a261-bce740046772","Type":"ContainerDied","Data":"99c3ca732312c8785f696e01bc56fc3827f09f2797e98f65e758b381ef31258d"} Oct 30 17:42:13.818514 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.818488 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9" event={"ID":"b32fbfdf-cbf9-4cf3-a261-bce740046772","Type":"ContainerDied","Data":"85a98911b4f9e6dd8c558ce73818969d4f1f94798a74bf4e7b9cc7abe7277c1b"} Oct 30 17:42:13.833787 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.833757 4828 scope.go:122] "RemoveContainer" containerID="99c3ca732312c8785f696e01bc56fc3827f09f2797e98f65e758b381ef31258d" Oct 30 17:42:13.862357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.862336 4828 scope.go:122] "RemoveContainer" containerID="99c3ca732312c8785f696e01bc56fc3827f09f2797e98f65e758b381ef31258d" Oct 30 17:42:13.862636 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:42:13.862618 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c3ca732312c8785f696e01bc56fc3827f09f2797e98f65e758b381ef31258d\": container with ID starting with 99c3ca732312c8785f696e01bc56fc3827f09f2797e98f65e758b381ef31258d not found: ID does not exist" containerID="99c3ca732312c8785f696e01bc56fc3827f09f2797e98f65e758b381ef31258d" Oct 30 17:42:13.862689 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.862646 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c3ca732312c8785f696e01bc56fc3827f09f2797e98f65e758b381ef31258d"} err="failed to get container status \"99c3ca732312c8785f696e01bc56fc3827f09f2797e98f65e758b381ef31258d\": rpc error: code = NotFound desc = could not find container \"99c3ca732312c8785f696e01bc56fc3827f09f2797e98f65e758b381ef31258d\": container with ID starting with 99c3ca732312c8785f696e01bc56fc3827f09f2797e98f65e758b381ef31258d not found: ID does not exist" Oct 30 17:42:13.872400 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.872374 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9"] Oct 30 17:42:13.884973 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.884944 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xxzj9"] Oct 30 17:42:13.911839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.911759 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78487d6d64-fkx6f"] Oct 30 17:42:13.916293 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:13.916267 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78487d6d64-fkx6f"] Oct 30 17:42:14.488591 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:14.488552 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32fbfdf-cbf9-4cf3-a261-bce740046772" path="/var/lib/kubelet/pods/b32fbfdf-cbf9-4cf3-a261-bce740046772/volumes" Oct 30 17:42:14.488967 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:14.488954 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0507da-9433-43b6-9f30-be94f7e1cf06" path="/var/lib/kubelet/pods/ec0507da-9433-43b6-9f30-be94f7e1cf06/volumes" Oct 30 17:42:14.954113 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:14.954076 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:42:15.118316 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.118281 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78487d6d64-k525h"] Oct 30 17:42:15.118702 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.118690 4828 cpu_manager.go:414] "containerMap: removing container" podUID="ec0507da-9433-43b6-9f30-be94f7e1cf06" containerName="manager" Oct 30 17:42:15.118754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.118703 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0507da-9433-43b6-9f30-be94f7e1cf06" containerName="manager" Oct 30 17:42:15.118754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.118741 4828 cpu_manager.go:414] "containerMap: removing container" podUID="b32fbfdf-cbf9-4cf3-a261-bce740046772" containerName="discovery" Oct 30 17:42:15.118754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.118747 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32fbfdf-cbf9-4cf3-a261-bce740046772" containerName="discovery" Oct 30 17:42:15.118868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.118822 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32fbfdf-cbf9-4cf3-a261-bce740046772" containerName="discovery" Oct 30 17:42:15.118868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.118830 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0507da-9433-43b6-9f30-be94f7e1cf06" containerName="manager" Oct 30 17:42:15.123486 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.123460 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78487d6d64-k525h" Oct 30 17:42:15.138878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.138857 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 17:42:15.151434 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.151381 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 17:42:15.151574 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.151476 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8xm5f\"" Oct 30 17:42:15.151574 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.151477 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 17:42:15.154277 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.154252 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78487d6d64-k525h"] Oct 30 17:42:15.182913 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.182868 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7z7\" (UniqueName: \"kubernetes.io/projected/b8f04c76-eb20-4b7a-8896-e09db947337d-kube-api-access-qh7z7\") pod \"maas-controller-78487d6d64-k525h\" (UID: \"b8f04c76-eb20-4b7a-8896-e09db947337d\") " pod="opendatahub/maas-controller-78487d6d64-k525h" Oct 30 17:42:15.183084 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.182940 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8f04c76-eb20-4b7a-8896-e09db947337d-webhook-cert\") pod \"maas-controller-78487d6d64-k525h\" (UID: \"b8f04c76-eb20-4b7a-8896-e09db947337d\") " pod="opendatahub/maas-controller-78487d6d64-k525h" Oct 30 17:42:15.183084 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.183047 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f04c76-eb20-4b7a-8896-e09db947337d-metrics-certs\") pod \"maas-controller-78487d6d64-k525h\" (UID: \"b8f04c76-eb20-4b7a-8896-e09db947337d\") " pod="opendatahub/maas-controller-78487d6d64-k525h" Oct 30 17:42:15.284642 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.284601 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7z7\" (UniqueName: \"kubernetes.io/projected/b8f04c76-eb20-4b7a-8896-e09db947337d-kube-api-access-qh7z7\") pod \"maas-controller-78487d6d64-k525h\" (UID: \"b8f04c76-eb20-4b7a-8896-e09db947337d\") " pod="opendatahub/maas-controller-78487d6d64-k525h" Oct 30 17:42:15.284799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.284682 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8f04c76-eb20-4b7a-8896-e09db947337d-webhook-cert\") pod \"maas-controller-78487d6d64-k525h\" (UID: \"b8f04c76-eb20-4b7a-8896-e09db947337d\") " pod="opendatahub/maas-controller-78487d6d64-k525h" Oct 30 17:42:15.284799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.284712 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f04c76-eb20-4b7a-8896-e09db947337d-metrics-certs\") pod \"maas-controller-78487d6d64-k525h\" (UID: \"b8f04c76-eb20-4b7a-8896-e09db947337d\") " pod="opendatahub/maas-controller-78487d6d64-k525h" Oct 30 17:42:15.287440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.287412 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8f04c76-eb20-4b7a-8896-e09db947337d-metrics-certs\") pod \"maas-controller-78487d6d64-k525h\" (UID: \"b8f04c76-eb20-4b7a-8896-e09db947337d\") " pod="opendatahub/maas-controller-78487d6d64-k525h" Oct 30 17:42:15.287553 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.287422 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b8f04c76-eb20-4b7a-8896-e09db947337d-webhook-cert\") pod \"maas-controller-78487d6d64-k525h\" (UID: \"b8f04c76-eb20-4b7a-8896-e09db947337d\") " pod="opendatahub/maas-controller-78487d6d64-k525h" Oct 30 17:42:15.309020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.308983 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7z7\" (UniqueName: \"kubernetes.io/projected/b8f04c76-eb20-4b7a-8896-e09db947337d-kube-api-access-qh7z7\") pod \"maas-controller-78487d6d64-k525h\" (UID: \"b8f04c76-eb20-4b7a-8896-e09db947337d\") " pod="opendatahub/maas-controller-78487d6d64-k525h" Oct 30 17:42:15.436233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.436193 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78487d6d64-k525h" Oct 30 17:42:15.737806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.737777 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78487d6d64-k525h"] Oct 30 17:42:15.738796 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:42:15.738764 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f04c76_eb20_4b7a_8896_e09db947337d.slice/crio-b9ca00e06b723ba3893bf467da9501213e3342cbd4a8ae9be0b649222b4d9e00 WatchSource:0}: Error finding container b9ca00e06b723ba3893bf467da9501213e3342cbd4a8ae9be0b649222b4d9e00: Status 404 returned error can't find the container with id b9ca00e06b723ba3893bf467da9501213e3342cbd4a8ae9be0b649222b4d9e00 Oct 30 17:42:15.832363 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:15.832324 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78487d6d64-k525h" event={"ID":"b8f04c76-eb20-4b7a-8896-e09db947337d","Type":"ContainerStarted","Data":"b9ca00e06b723ba3893bf467da9501213e3342cbd4a8ae9be0b649222b4d9e00"} Oct 30 17:42:16.840250 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:16.840214 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78487d6d64-k525h" event={"ID":"b8f04c76-eb20-4b7a-8896-e09db947337d","Type":"ContainerStarted","Data":"78f499fd0ad2b96ea7dced64ffaa495ea5e30c0b0e490c91d9ee8ec83d063a37"} Oct 30 17:42:16.840682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:16.840290 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78487d6d64-k525h" Oct 30 17:42:16.885126 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:16.885072 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78487d6d64-k525h" podStartSLOduration=1.562337906 podStartE2EDuration="1.885059589s" totalImagesPullingTime="322.721683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:42:16.884601854 +0000 UTC m=+3314.530112035" watchObservedRunningTime="2026-10-30 17:42:16.885059589 +0000 UTC m=+3314.530569769" Oct 30 17:42:18.387649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:18.387607 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:42:27.848450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:27.848415 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78487d6d64-k525h" Oct 30 17:42:33.926547 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:33.926512 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:42:34.083180 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:34.083140 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:42:37.910978 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:42:37.910943 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:42:38.337013 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:42:38.336972 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:43:01.068445 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:01.068411 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:43:01.684127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:01.684086 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:43:07.912345 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:43:07.912297 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:43:09.884444 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:09.884376 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7"] Oct 30 17:43:09.906317 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:09.906247 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:09.954493 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:09.954449 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb"] Oct 30 17:43:09.962495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:09.962042 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:09.962495 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:09.962467 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7"] Oct 30 17:43:09.962659 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:09.962523 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb"] Oct 30 17:43:10.051370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.051326 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ac60a7-ad93-4811-8242-5f28289f5d1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.051370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.051375 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ac60a7-ad93-4811-8242-5f28289f5d1c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.051622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.051454 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.051622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.051580 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ac60a7-ad93-4811-8242-5f28289f5d1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.051622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.051605 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.051778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.051625 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.051778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.051649 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.051778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.051749 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.051911 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.051797 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8sq\" (UniqueName: \"kubernetes.io/projected/54ac60a7-ad93-4811-8242-5f28289f5d1c-kube-api-access-vv8sq\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.153209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153107 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.153209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153169 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ac60a7-ad93-4811-8242-5f28289f5d1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.153477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153252 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.153477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153298 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.153477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153338 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.153477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153378 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.153477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153406 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.153477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153445 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.153477 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153473 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.153868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153500 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.153868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153545 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.153868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153592 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8sq\" (UniqueName: \"kubernetes.io/projected/54ac60a7-ad93-4811-8242-5f28289f5d1c-kube-api-access-vv8sq\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.153868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153633 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ac60a7-ad93-4811-8242-5f28289f5d1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.153868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153665 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ac60a7-ad93-4811-8242-5f28289f5d1c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.153868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153684 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.153868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153691 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nvwh\" (UniqueName: \"kubernetes.io/projected/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-kube-api-access-8nvwh\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.154271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153950 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.154271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153993 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.154271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.153999 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.154435 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.154294 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.154435 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.154384 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.154435 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.154388 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.154763 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.154732 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54ac60a7-ad93-4811-8242-5f28289f5d1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.156142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.156121 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54ac60a7-ad93-4811-8242-5f28289f5d1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.156882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.156861 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54ac60a7-ad93-4811-8242-5f28289f5d1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.190357 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.190290 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54ac60a7-ad93-4811-8242-5f28289f5d1c-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.213850 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.213773 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8sq\" (UniqueName: \"kubernetes.io/projected/54ac60a7-ad93-4811-8242-5f28289f5d1c-kube-api-access-vv8sq\") pod \"maas-default-gateway-openshift-default-6db586dd6c-22qj7\" (UID: \"54ac60a7-ad93-4811-8242-5f28289f5d1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.255665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.255620 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.255887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.255687 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.255887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.255724 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.255887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.255747 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.255887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.255783 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.255887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.255841 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nvwh\" (UniqueName: \"kubernetes.io/projected/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-kube-api-access-8nvwh\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.255887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.255874 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.256299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.255921 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.256299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.255946 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.256419 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.256406 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.256860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.256800 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.256984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.256932 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.256984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.256978 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.257268 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.257243 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.258820 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.258792 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.259046 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.259027 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.288934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.288461 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:10.330258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.312891 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.330258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.314648 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nvwh\" (UniqueName: \"kubernetes.io/projected/aaebd75a-5133-4ca6-b3f9-9adfd87dd67d-kube-api-access-8nvwh\") pod \"maas-default-gateway-openshift-default-6db586dd6c-xmtgb\" (UID: \"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.607595 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.607269 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:10.712785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.708498 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7"] Oct 30 17:43:10.719669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.718713 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:43:10.719669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.718779 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:43:10.719669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.718808 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:43:10.939137 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:43:10.939095 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaebd75a_5133_4ca6_b3f9_9adfd87dd67d.slice/crio-b08679f2a7b23983fd6a29e2c7370c61f1922ad05a530ed8850ef670a888fc29 WatchSource:0}: Error finding container b08679f2a7b23983fd6a29e2c7370c61f1922ad05a530ed8850ef670a888fc29: Status 404 returned error can't find the container with id b08679f2a7b23983fd6a29e2c7370c61f1922ad05a530ed8850ef670a888fc29 Oct 30 17:43:10.941824 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.941782 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:43:10.942047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.941859 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:43:10.942047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.941983 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:43:10.942561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:10.942534 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb"] Oct 30 17:43:11.151427 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:11.151321 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" event={"ID":"54ac60a7-ad93-4811-8242-5f28289f5d1c","Type":"ContainerStarted","Data":"7f28f0f3436bfc442caaa2a14449b131cc0adc7d2f3d01996956678670fdcc5c"} Oct 30 17:43:11.151427 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:11.151388 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" event={"ID":"54ac60a7-ad93-4811-8242-5f28289f5d1c","Type":"ContainerStarted","Data":"d0f6e39480ead5b8cc96ba275846f875b42e195dc9d39b14d76dedab577ceb25"} Oct 30 17:43:11.153214 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:11.153184 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" event={"ID":"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d","Type":"ContainerStarted","Data":"85c0f2aabb2ac6419c4306ae3f91c815356e82b34720748ba34471529f98acdb"} Oct 30 17:43:11.153337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:11.153240 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" event={"ID":"aaebd75a-5133-4ca6-b3f9-9adfd87dd67d","Type":"ContainerStarted","Data":"b08679f2a7b23983fd6a29e2c7370c61f1922ad05a530ed8850ef670a888fc29"} Oct 30 17:43:11.252059 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:11.251930 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" podStartSLOduration=2.251877374 podStartE2EDuration="2.251877374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:11.203976213 +0000 UTC m=+3368.849486393" watchObservedRunningTime="2026-10-30 17:43:11.251877374 +0000 UTC m=+3368.897387555" Oct 30 17:43:11.254218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:11.254177 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" podStartSLOduration=2.2541661 podStartE2EDuration="2.2541661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:43:11.251026889 +0000 UTC m=+3368.896537218" watchObservedRunningTime="2026-10-30 17:43:11.2541661 +0000 UTC m=+3368.899676338" Oct 30 17:43:11.292375 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:11.292288 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:11.296346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:11.296014 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-22qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:43:11.296346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:11.296073 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" podUID="54ac60a7-ad93-4811-8242-5f28289f5d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:43:11.614924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:11.614774 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:12.304427 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:12.304383 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-22qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:43:12.305066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:12.305026 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" podUID="54ac60a7-ad93-4811-8242-5f28289f5d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:43:12.623134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:12.623031 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xmtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 17:43:12.623134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:12.623099 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" podUID="aaebd75a-5133-4ca6-b3f9-9adfd87dd67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 17:43:13.289464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:13.289420 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-22qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:43:13.289798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:13.289767 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" podUID="54ac60a7-ad93-4811-8242-5f28289f5d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:43:13.618926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:13.618366 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xmtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 17:43:13.618926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:13.618434 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" podUID="aaebd75a-5133-4ca6-b3f9-9adfd87dd67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 17:43:14.292932 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:14.292669 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-22qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:43:14.293725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:14.293196 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" podUID="54ac60a7-ad93-4811-8242-5f28289f5d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:43:14.610981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:14.610403 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xmtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 17:43:14.610981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:14.610470 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" podUID="aaebd75a-5133-4ca6-b3f9-9adfd87dd67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 17:43:15.290358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:15.290312 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-22qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:43:15.290994 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:15.290963 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" podUID="54ac60a7-ad93-4811-8242-5f28289f5d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:43:15.614828 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:15.614731 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xmtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 17:43:15.615117 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:15.615088 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" podUID="aaebd75a-5133-4ca6-b3f9-9adfd87dd67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 17:43:16.292925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:16.289332 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-22qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:43:16.292925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:16.289407 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" podUID="54ac60a7-ad93-4811-8242-5f28289f5d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:43:16.618425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:16.618323 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xmtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 17:43:16.618701 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:16.618673 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" podUID="aaebd75a-5133-4ca6-b3f9-9adfd87dd67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 17:43:17.289665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:17.289614 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-22qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:43:17.289874 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:17.289684 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" podUID="54ac60a7-ad93-4811-8242-5f28289f5d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:43:17.610954 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:17.610829 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xmtgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" start-of-body= Oct 30 17:43:17.610954 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:17.610892 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" podUID="aaebd75a-5133-4ca6-b3f9-9adfd87dd67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": dial tcp 10.132.0.245:15021: connect: connection refused" Oct 30 17:43:18.289083 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:18.289039 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-22qj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Oct 30 17:43:18.289304 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:18.289094 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" podUID="54ac60a7-ad93-4811-8242-5f28289f5d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Oct 30 17:43:19.293963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:19.293922 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:19.294447 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:19.294347 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:19.295045 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:19.295019 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" Oct 30 17:43:19.340552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:19.340517 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:19.340966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:19.340939 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:19.341958 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:19.341940 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" Oct 30 17:43:21.843753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:21.843674 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:43:22.015163 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:43:22.015123 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:43:37.910761 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:43:37.910721 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:44:01.236108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:44:01.236066 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:44:05.341924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:44:05.341021 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-22qj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:44:05.341924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:44:05.341115 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" podUID="54ac60a7-ad93-4811-8242-5f28289f5d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:44:05.385062 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:44:05.385015 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xmtgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:44:05.385404 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:44:05.385364 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" podUID="aaebd75a-5133-4ca6-b3f9-9adfd87dd67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:44:07.910003 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:44:07.909961 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:44:20.344070 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:44:20.344021 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-22qj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:44:20.344845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:44:20.344810 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" podUID="54ac60a7-ad93-4811-8242-5f28289f5d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:44:20.407350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:44:20.407306 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xmtgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:44:20.408314 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:44:20.408274 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" podUID="aaebd75a-5133-4ca6-b3f9-9adfd87dd67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:44:37.912711 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:44:37.912664 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:45:01.308118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:45:01.308079 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:45:07.913012 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:45:07.912972 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:45:20.355131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:45:20.355054 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-22qj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:45:20.416338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:45:20.367054 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" podUID="54ac60a7-ad93-4811-8242-5f28289f5d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:45:20.440134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:45:20.416283 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-xmtgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:45:20.441314 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:45:20.441265 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-xmtgb" podUID="aaebd75a-5133-4ca6-b3f9-9adfd87dd67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:45:34.560728 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:45:34.560688 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:45:37.910384 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:45:37.910350 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:46:01.436454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:01.436405 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:04.962292 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:04.962250 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:07.910869 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:46:07.910832 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:46:13.032368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:13.032334 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq"] Oct 30 17:46:13.043557 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:13.043524 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-627571c3-2l4nq"] Oct 30 17:46:14.490325 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:14.490231 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5322d132-73a0-45e0-a7f8-0a82949e8ff2" path="/var/lib/kubelet/pods/5322d132-73a0-45e0-a7f8-0a82949e8ff2/volumes" Oct 30 17:46:37.912083 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:46:37.912042 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:46:40.257532 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.257491 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:40.778393 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.778357 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t"] Oct 30 17:46:40.782125 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.782081 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.796529 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.796500 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-c118c3-openshift-default-dockercfg-xfn9b\"" Oct 30 17:46:40.811654 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.811616 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t"] Oct 30 17:46:40.851564 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.851531 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-data\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.851726 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.851569 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-envoy\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.851726 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.851590 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz258\" (UniqueName: \"kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-kube-api-access-nz258\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.851726 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.851714 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istiod-ca-cert\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.851937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.851746 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-token\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.852008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.851934 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-socket\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.852008 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.851984 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-podinfo\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.852110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.852019 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-credential-socket\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.852110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.852048 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-certs\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.883561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.883530 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:40.952885 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.952851 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-socket\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.953080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.952891 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-podinfo\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.953080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.952960 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-credential-socket\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.953080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.952988 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-certs\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.953080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.953022 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-data\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.953274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.953232 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-envoy\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.953337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.953280 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz258\" (UniqueName: \"kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-kube-api-access-nz258\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.953337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.953300 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-socket\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.953450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.953350 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-data\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.953450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.953360 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istiod-ca-cert\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.953450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.953412 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-token\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.953450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.953417 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-certs\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.953682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.953454 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-credential-socket\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.954050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.954027 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istiod-ca-cert\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.955541 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.955518 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-envoy\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.955693 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.955673 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-podinfo\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.989865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.989832 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz258\" (UniqueName: \"kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-kube-api-access-nz258\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:40.994700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:40.994677 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-token\") pod \"e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:41.096762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:41.096661 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:41.122624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:41.122586 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:41.432670 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:41.432635 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t"] Oct 30 17:46:41.433596 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:46:41.433567 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d7a6a4_aa78_4aa3_b31d_05682ec67e03.slice/crio-2b0bd2af97963ac0fc8c587034d5a19414b2a1ae857eb05044ff77635de5843a WatchSource:0}: Error finding container 2b0bd2af97963ac0fc8c587034d5a19414b2a1ae857eb05044ff77635de5843a: Status 404 returned error can't find the container with id 2b0bd2af97963ac0fc8c587034d5a19414b2a1ae857eb05044ff77635de5843a Oct 30 17:46:41.435968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:41.435930 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:46:41.436053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:41.436017 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:46:41.436108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:41.436093 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:46:42.190985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:42.190942 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" event={"ID":"95d7a6a4-aa78-4aa3-b31d-05682ec67e03","Type":"ContainerStarted","Data":"200fd1bb16fb79d18632e33f67f870cf880512356cf6f1826657bfdea62b5178"} Oct 30 17:46:42.191182 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:42.191011 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" event={"ID":"95d7a6a4-aa78-4aa3-b31d-05682ec67e03","Type":"ContainerStarted","Data":"2b0bd2af97963ac0fc8c587034d5a19414b2a1ae857eb05044ff77635de5843a"} Oct 30 17:46:42.250455 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:42.250391 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" podStartSLOduration=2.250371448 podStartE2EDuration="2.250371448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:42.248233495 +0000 UTC m=+3579.893743675" watchObservedRunningTime="2026-10-30 17:46:42.250371448 +0000 UTC m=+3579.895881629" Oct 30 17:46:43.097675 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:43.097616 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:43.104162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:43.104129 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:43.206202 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:43.206163 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:43.207430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:43.207403 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:46:44.877459 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:44.877416 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:45.032406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:45.032347 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:46.365136 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.365099 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9"] Oct 30 17:46:46.368938 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.368892 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:46.384103 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.384075 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-c118c3-serving-cert\"" Oct 30 17:46:46.398890 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.398851 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9"] Oct 30 17:46:46.417604 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.417564 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815eeede-5d37-4219-8a39-8b122253ced2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:46.417829 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.417745 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwc5v\" (UniqueName: \"kubernetes.io/projected/815eeede-5d37-4219-8a39-8b122253ced2-kube-api-access-bwc5v\") pod \"maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:46.417891 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.417858 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:46.418025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.418006 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:46.519183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.519143 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815eeede-5d37-4219-8a39-8b122253ced2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:46.519431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.519213 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwc5v\" (UniqueName: \"kubernetes.io/projected/815eeede-5d37-4219-8a39-8b122253ced2-kube-api-access-bwc5v\") pod \"maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:46.519431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.519254 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:46.519431 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.519311 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:46.519431 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:46:46.519410 4828 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-serving-cert: secret "maas-api-e2e-worker-single-main-c118c3-serving-cert" not found Oct 30 17:46:46.519753 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:46:46.519528 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-maas-api-tls podName:815eeede-5d37-4219-8a39-8b122253ced2 nodeName:}" failed. No retries permitted until 2026-10-30 17:46:47.019504994 +0000 UTC m=+3584.665015169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-maas-api-tls") pod "maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" (UID: "815eeede-5d37-4219-8a39-8b122253ced2") : secret "maas-api-e2e-worker-single-main-c118c3-serving-cert" not found Oct 30 17:46:46.519951 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.519925 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815eeede-5d37-4219-8a39-8b122253ced2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:46.522301 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.522273 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:46.542638 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.542600 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwc5v\" (UniqueName: \"kubernetes.io/projected/815eeede-5d37-4219-8a39-8b122253ced2-kube-api-access-bwc5v\") pod \"maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:46.548570 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.548535 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:46.699882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.699775 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:46.728694 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.728657 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g"] Oct 30 17:46:46.732464 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.732443 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" Oct 30 17:46:46.752011 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.751987 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-c118c3\"" Oct 30 17:46:46.752500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.752480 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b61db25e-dockercfg-tpmzm\"" Oct 30 17:46:46.756090 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.756064 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g"] Oct 30 17:46:46.822867 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.822815 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wwnp\" (UniqueName: \"kubernetes.io/projected/a913e1db-f066-4ce7-bf78-46fd2f584436-kube-api-access-9wwnp\") pod \"payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g\" (UID: \"a913e1db-f066-4ce7-bf78-46fd2f584436\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" Oct 30 17:46:46.823086 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.822986 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a913e1db-f066-4ce7-bf78-46fd2f584436-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g\" (UID: \"a913e1db-f066-4ce7-bf78-46fd2f584436\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" Oct 30 17:46:46.825980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.825947 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99"] Oct 30 17:46:46.830359 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.830334 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" Oct 30 17:46:46.850704 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.850676 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99"] Oct 30 17:46:46.923806 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.923769 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgpvp\" (UniqueName: \"kubernetes.io/projected/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-kube-api-access-qgpvp\") pod \"payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99\" (UID: \"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" Oct 30 17:46:46.924000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.923818 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99\" (UID: \"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" Oct 30 17:46:46.924000 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.923940 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wwnp\" (UniqueName: \"kubernetes.io/projected/a913e1db-f066-4ce7-bf78-46fd2f584436-kube-api-access-9wwnp\") pod \"payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g\" (UID: \"a913e1db-f066-4ce7-bf78-46fd2f584436\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" Oct 30 17:46:46.924098 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.924036 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a913e1db-f066-4ce7-bf78-46fd2f584436-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g\" (UID: \"a913e1db-f066-4ce7-bf78-46fd2f584436\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" Oct 30 17:46:46.924706 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.924676 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a913e1db-f066-4ce7-bf78-46fd2f584436-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g\" (UID: \"a913e1db-f066-4ce7-bf78-46fd2f584436\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" Oct 30 17:46:46.947120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:46.947085 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wwnp\" (UniqueName: \"kubernetes.io/projected/a913e1db-f066-4ce7-bf78-46fd2f584436-kube-api-access-9wwnp\") pod \"payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g\" (UID: \"a913e1db-f066-4ce7-bf78-46fd2f584436\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" Oct 30 17:46:47.025362 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:47.025324 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgpvp\" (UniqueName: \"kubernetes.io/projected/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-kube-api-access-qgpvp\") pod \"payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99\" (UID: \"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" Oct 30 17:46:47.025577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:47.025379 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99\" (UID: \"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" Oct 30 17:46:47.025577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:47.025431 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:47.026223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:47.026195 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99\" (UID: \"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" Oct 30 17:46:47.027849 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:47.027828 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:47.045977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:47.045949 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" Oct 30 17:46:47.048454 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:47.048428 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgpvp\" (UniqueName: \"kubernetes.io/projected/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-kube-api-access-qgpvp\") pod \"payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99\" (UID: \"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" Oct 30 17:46:47.150604 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:47.150569 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" Oct 30 17:46:47.283596 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:47.283497 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:47.352271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:47.352233 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g"] Oct 30 17:46:47.353952 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:46:47.353924 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda913e1db_f066_4ce7_bf78_46fd2f584436.slice/crio-9f5b87e038c13cf4b3ca9b0a66336244cecfe10803056a0a7e7af1db4893de89 WatchSource:0}: Error finding container 9f5b87e038c13cf4b3ca9b0a66336244cecfe10803056a0a7e7af1db4893de89: Status 404 returned error can't find the container with id 9f5b87e038c13cf4b3ca9b0a66336244cecfe10803056a0a7e7af1db4893de89 Oct 30 17:46:47.463360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:47.463329 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99"] Oct 30 17:46:47.464838 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:46:47.464813 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6d1441_5515_4a66_b9e5_4811f2ff7a6a.slice/crio-329a35ebe82f69e466b0ae405e6d114dfa76bdd112baa6e7509a5eadc2640a83 WatchSource:0}: Error finding container 329a35ebe82f69e466b0ae405e6d114dfa76bdd112baa6e7509a5eadc2640a83: Status 404 returned error can't find the container with id 329a35ebe82f69e466b0ae405e6d114dfa76bdd112baa6e7509a5eadc2640a83 Oct 30 17:46:47.590219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:47.590186 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9"] Oct 30 17:46:47.591879 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:46:47.591847 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815eeede_5d37_4219_8a39_8b122253ced2.slice/crio-a1a6b5637aa8e6772b9342c6be53567cb64f2b42e1d7cdf2a71e7b8b6a1de8f1 WatchSource:0}: Error finding container a1a6b5637aa8e6772b9342c6be53567cb64f2b42e1d7cdf2a71e7b8b6a1de8f1: Status 404 returned error can't find the container with id a1a6b5637aa8e6772b9342c6be53567cb64f2b42e1d7cdf2a71e7b8b6a1de8f1 Oct 30 17:46:47.593480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:47.593461 4828 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:46:48.238295 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:48.238186 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" event={"ID":"815eeede-5d37-4219-8a39-8b122253ced2","Type":"ContainerStarted","Data":"ba6a420812eef20c27daed60a77a8b7e22418fe882bfe051c50290963e3cdb41"} Oct 30 17:46:48.238295 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:48.238261 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:48.238295 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:48.238288 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" event={"ID":"815eeede-5d37-4219-8a39-8b122253ced2","Type":"ContainerStarted","Data":"a1a6b5637aa8e6772b9342c6be53567cb64f2b42e1d7cdf2a71e7b8b6a1de8f1"} Oct 30 17:46:48.241442 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:48.240236 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" event={"ID":"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a","Type":"ContainerStarted","Data":"a6586b22ecbd35ad47786d96ab83885ad6024fb833caa9ffa911d96234f5df9e"} Oct 30 17:46:48.241442 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:48.240297 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" Oct 30 17:46:48.241442 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:48.240319 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" event={"ID":"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a","Type":"ContainerStarted","Data":"329a35ebe82f69e466b0ae405e6d114dfa76bdd112baa6e7509a5eadc2640a83"} Oct 30 17:46:48.241969 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:48.241945 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" event={"ID":"a913e1db-f066-4ce7-bf78-46fd2f584436","Type":"ContainerStarted","Data":"c8eb291ab508a724e091e73048852f0fa17220b0dba354af7cab8f16bcdac59d"} Oct 30 17:46:48.242057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:48.241989 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" event={"ID":"a913e1db-f066-4ce7-bf78-46fd2f584436","Type":"ContainerStarted","Data":"9f5b87e038c13cf4b3ca9b0a66336244cecfe10803056a0a7e7af1db4893de89"} Oct 30 17:46:48.242117 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:48.242075 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" Oct 30 17:46:48.291913 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:48.291856 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" podStartSLOduration=1.97142423 podStartE2EDuration="2.291842159s" totalImagesPullingTime="320.417929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:48.290986803 +0000 UTC m=+3585.936496977" watchObservedRunningTime="2026-10-30 17:46:48.291842159 +0000 UTC m=+3585.937352338" Oct 30 17:46:48.334058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:48.333995 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" podStartSLOduration=2.333980294 podStartE2EDuration="2.333980294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:48.332566127 +0000 UTC m=+3585.978076307" watchObservedRunningTime="2026-10-30 17:46:48.333980294 +0000 UTC m=+3585.979490474" Oct 30 17:46:48.371565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:48.371523 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:48.377007 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:48.376944 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" podStartSLOduration=2.376928053 podStartE2EDuration="2.376928053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:46:48.376745998 +0000 UTC m=+3586.022256180" watchObservedRunningTime="2026-10-30 17:46:48.376928053 +0000 UTC m=+3586.022438233" Oct 30 17:46:50.163416 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:50.163379 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:52.116780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:52.116741 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:53.884622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:53.884581 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:54.253921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:54.253825 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:46:55.805926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:55.805867 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:57.674634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:57.674591 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:59.248101 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:59.248060 4828 patch_prober.go:29] interesting pod/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:46:59.248634 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:59.248146 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:46:59.252254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:59.252224 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" Oct 30 17:46:59.252405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:59.252314 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" Oct 30 17:46:59.483838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:59.483798 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:46:59.821349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:46:59.821309 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:00.342570 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:00.342530 4828 generic.go:356] "Generic (PLEG): container finished" podID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerID="200fd1bb16fb79d18632e33f67f870cf880512356cf6f1826657bfdea62b5178" exitCode=0 Oct 30 17:47:00.343116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:00.342605 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" event={"ID":"95d7a6a4-aa78-4aa3-b31d-05682ec67e03","Type":"ContainerDied","Data":"200fd1bb16fb79d18632e33f67f870cf880512356cf6f1826657bfdea62b5178"} Oct 30 17:47:00.343116 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:00.343048 4828 scope.go:122] "RemoveContainer" containerID="200fd1bb16fb79d18632e33f67f870cf880512356cf6f1826657bfdea62b5178" Oct 30 17:47:00.344302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:00.344275 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:47:00.344395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:00.344342 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:47:00.344395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:00.344376 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:47:00.465870 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:00.465720 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:00.659437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:00.659394 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:00.820374 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:00.820324 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:01.350648 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:01.350613 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" event={"ID":"95d7a6a4-aa78-4aa3-b31d-05682ec67e03","Type":"ContainerStarted","Data":"9f5e457c5564bab38602e9566e0da68b71650a52c9903b418f5c5a0f26858a52"} Oct 30 17:47:01.364649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:01.364608 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:01.595852 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:01.595812 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:02.097600 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:02.097566 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:47:02.097810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:02.097776 4828 patch_prober.go:29] interesting pod/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 17:47:02.097882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:02.097843 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 17:47:02.194767 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:02.194728 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:03.097274 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.097232 4828 patch_prober.go:29] interesting pod/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 17:47:03.097687 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.097288 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 17:47:03.131425 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.131386 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:03.501237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.501138 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z"] Oct 30 17:47:03.507506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.507468 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.509601 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.509563 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z"] Oct 30 17:47:03.533849 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.533815 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c118c3\"/\"default-dockercfg-xvfph\"" Oct 30 17:47:03.534052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.534000 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c118c3\"/\"kube-root-ca.crt\"" Oct 30 17:47:03.534052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.534031 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-c118c3\"/\"simulator-single-main-c118c3-kserve-self-signed-certs\"" Oct 30 17:47:03.534174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.534112 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-c118c3\"/\"openshift-service-ca.crt\"" Oct 30 17:47:03.596244 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.594342 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:03.624152 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.624112 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-home\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.624346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.624160 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-model-cache\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.624346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.624204 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsqwm\" (UniqueName: \"kubernetes.io/projected/d293ba72-71a4-4d3d-b559-82f1b009a5ef-kube-api-access-nsqwm\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.624346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.624263 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tmp-dir\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.624346 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.624285 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tls-certs\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.624596 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.624562 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-dshm\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.725887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.725851 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-dshm\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.726144 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.725972 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-home\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.726144 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.726001 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-model-cache\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.726144 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.726062 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsqwm\" (UniqueName: \"kubernetes.io/projected/d293ba72-71a4-4d3d-b559-82f1b009a5ef-kube-api-access-nsqwm\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.726144 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.726100 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tmp-dir\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.726144 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.726132 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tls-certs\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.727162 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.727105 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tmp-dir\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.727715 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.727658 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-home\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.728131 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.728096 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-model-cache\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.736218 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.736188 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tls-certs\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.736386 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.736365 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-dshm\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.749832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.749802 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsqwm\" (UniqueName: \"kubernetes.io/projected/d293ba72-71a4-4d3d-b559-82f1b009a5ef-kube-api-access-nsqwm\") pod \"simulator-single-main-c118c3-kserve-59df49bf85-v774z\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:03.823273 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:03.823236 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:04.097423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:04.097330 4828 patch_prober.go:29] interesting pod/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 17:47:04.097423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:04.097395 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 17:47:04.153066 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:04.153033 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z"] Oct 30 17:47:04.155259 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:47:04.155232 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd293ba72_71a4_4d3d_b559_82f1b009a5ef.slice/crio-134653c1cfedceb3903d376f9578f622097ba5ac167b256fc1ccf7bc2e767d75 WatchSource:0}: Error finding container 134653c1cfedceb3903d376f9578f622097ba5ac167b256fc1ccf7bc2e767d75: Status 404 returned error can't find the container with id 134653c1cfedceb3903d376f9578f622097ba5ac167b256fc1ccf7bc2e767d75 Oct 30 17:47:04.395271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:04.395175 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" event={"ID":"d293ba72-71a4-4d3d-b559-82f1b009a5ef","Type":"ContainerStarted","Data":"3875ae492a795c5f4316acb79b6f0bf9c037440ad672771ac88a2e2a247f25d6"} Oct 30 17:47:04.395271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:04.395223 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" event={"ID":"d293ba72-71a4-4d3d-b559-82f1b009a5ef","Type":"ContainerStarted","Data":"134653c1cfedceb3903d376f9578f622097ba5ac167b256fc1ccf7bc2e767d75"} Oct 30 17:47:04.438174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:04.438120 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" podStartSLOduration=1.438104693 podStartE2EDuration="1.438104693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:47:04.435693808 +0000 UTC m=+3602.081203988" watchObservedRunningTime="2026-10-30 17:47:04.438104693 +0000 UTC m=+3602.083614874" Oct 30 17:47:04.823576 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:04.823535 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:04.941265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:04.941221 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:05.098982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:05.098361 4828 patch_prober.go:29] interesting pod/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 17:47:05.098982 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:05.098429 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 17:47:06.098972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:06.097108 4828 patch_prober.go:29] interesting pod/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 30 17:47:06.098972 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:06.097169 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 30 17:47:06.659831 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:06.659785 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:07.102110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:07.102074 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:47:07.102569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:07.102483 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:47:07.103304 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:07.103281 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:47:07.911378 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:47:07.911338 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:47:08.024497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:08.024469 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:47:08.030345 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:08.030316 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:47:08.467539 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:08.467500 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:10.167704 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:10.167647 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:11.927870 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:11.927828 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:13.263440 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:13.263405 4828 scope.go:122] "RemoveContainer" containerID="4c2405e900a055bd10c8ff048cc6561e8ac75fa1b79c7b4752c717ac7a58aedb" Oct 30 17:47:13.273004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:13.272981 4828 scope.go:122] "RemoveContainer" containerID="47c697274b60990be99d3e1090e8a66eeaa488383ccd7b4f2c420c90694d24d3" Oct 30 17:47:13.282894 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:13.282871 4828 scope.go:122] "RemoveContainer" containerID="7bb2bb7365fdb134ff053ca9885f5a6c3b515617a02d24a0253810d691494702" Oct 30 17:47:13.572132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:13.571987 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:13.823630 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:13.823544 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:13.836739 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:13.836705 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:14.479036 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:14.479004 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:15.273430 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:15.273384 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:16.869404 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:16.869366 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:18.761777 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:18.761694 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:20.563265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:20.563213 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:22.449786 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:22.449745 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:24.101854 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:24.101815 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:25.772573 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:25.772531 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:27.427883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:27.427844 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:29.073273 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:29.073211 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:30.783546 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:30.783511 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:32.514965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:32.514916 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:34.285985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:34.285868 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:35.942051 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:35.942013 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:36.917993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:36.917950 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:37.761355 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:37.761319 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:37.912244 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:47:37.912206 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:47:39.437749 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:39.437680 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:41.225442 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:41.225393 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:41.833551 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:41.833516 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z"] Oct 30 17:47:41.944979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:41.944935 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:42.890597 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:42.890436 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:44.609688 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:44.609643 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:46.261716 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:46.261659 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:47.167085 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:47.167047 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99"] Oct 30 17:47:47.211093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:47.211055 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g"] Oct 30 17:47:47.366753 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:47.366719 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:48.527348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:48.527307 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:48.687314 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:48.687257 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:49.340303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.340265 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9"] Oct 30 17:47:49.340615 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.340576 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" podUID="815eeede-5d37-4219-8a39-8b122253ced2" containerName="maas-api" containerID="cri-o://ba6a420812eef20c27daed60a77a8b7e22418fe882bfe051c50290963e3cdb41" gracePeriod=30 Oct 30 17:47:49.461509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.461472 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:49.679650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.679619 4828 generic.go:356] "Generic (PLEG): container finished" podID="815eeede-5d37-4219-8a39-8b122253ced2" containerID="ba6a420812eef20c27daed60a77a8b7e22418fe882bfe051c50290963e3cdb41" exitCode=0 Oct 30 17:47:49.680040 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.679667 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" event={"ID":"815eeede-5d37-4219-8a39-8b122253ced2","Type":"ContainerDied","Data":"ba6a420812eef20c27daed60a77a8b7e22418fe882bfe051c50290963e3cdb41"} Oct 30 17:47:49.825467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.825433 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:47:49.900680 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.900576 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/815eeede-5d37-4219-8a39-8b122253ced2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815eeede-5d37-4219-8a39-8b122253ced2-openshift-service-ca\") pod \"815eeede-5d37-4219-8a39-8b122253ced2\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " Oct 30 17:47:49.900971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.900742 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815eeede-5d37-4219-8a39-8b122253ced2-kube-api-access-bwc5v\" (UniqueName: \"kubernetes.io/projected/815eeede-5d37-4219-8a39-8b122253ced2-kube-api-access-bwc5v\") pod \"815eeede-5d37-4219-8a39-8b122253ced2\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " Oct 30 17:47:49.900971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.900807 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-metrics-certs\") pod \"815eeede-5d37-4219-8a39-8b122253ced2\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " Oct 30 17:47:49.900971 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.900934 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-maas-api-tls\") pod \"815eeede-5d37-4219-8a39-8b122253ced2\" (UID: \"815eeede-5d37-4219-8a39-8b122253ced2\") " Oct 30 17:47:49.901209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.901060 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815eeede-5d37-4219-8a39-8b122253ced2-openshift-service-ca" pod "815eeede-5d37-4219-8a39-8b122253ced2" (UID: "815eeede-5d37-4219-8a39-8b122253ced2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:47:49.901340 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.901314 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815eeede-5d37-4219-8a39-8b122253ced2-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:47:49.903137 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.903112 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-maas-api-tls" pod "815eeede-5d37-4219-8a39-8b122253ced2" (UID: "815eeede-5d37-4219-8a39-8b122253ced2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:47:49.903337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.903306 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-metrics-certs" pod "815eeede-5d37-4219-8a39-8b122253ced2" (UID: "815eeede-5d37-4219-8a39-8b122253ced2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:47:49.903497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.903478 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815eeede-5d37-4219-8a39-8b122253ced2-kube-api-access-bwc5v" pod "815eeede-5d37-4219-8a39-8b122253ced2" (UID: "815eeede-5d37-4219-8a39-8b122253ced2"). InnerVolumeSpecName "kube-api-access-bwc5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:47:49.956880 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:49.956838 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:50.002339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:50.002302 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:47:50.002339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:50.002337 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815eeede-5d37-4219-8a39-8b122253ced2-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:47:50.002551 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:50.002351 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwc5v\" (UniqueName: \"kubernetes.io/projected/815eeede-5d37-4219-8a39-8b122253ced2-kube-api-access-bwc5v\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:47:50.107435 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:50.107397 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:50.624801 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:50.624756 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:50.688684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:50.688648 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" Oct 30 17:47:50.688684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:50.688676 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9" event={"ID":"815eeede-5d37-4219-8a39-8b122253ced2","Type":"ContainerDied","Data":"a1a6b5637aa8e6772b9342c6be53567cb64f2b42e1d7cdf2a71e7b8b6a1de8f1"} Oct 30 17:47:50.689282 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:50.688740 4828 scope.go:122] "RemoveContainer" containerID="ba6a420812eef20c27daed60a77a8b7e22418fe882bfe051c50290963e3cdb41" Oct 30 17:47:50.742023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:50.741985 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9"] Oct 30 17:47:50.753179 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:50.753086 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-c118c3-868bb68786-nvrx9"] Oct 30 17:47:52.168279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:52.168185 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" podUID="6a6d1441-5515-4a66-b9e5-4811f2ff7a6a" containerName="payload-processing" containerID="cri-o://a6586b22ecbd35ad47786d96ab83885ad6024fb833caa9ffa911d96234f5df9e" gracePeriod=25 Oct 30 17:47:52.211894 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:52.211830 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" podUID="a913e1db-f066-4ce7-bf78-46fd2f584436" containerName="payload-pre-processing" containerID="cri-o://c8eb291ab508a724e091e73048852f0fa17220b0dba354af7cab8f16bcdac59d" gracePeriod=25 Oct 30 17:47:52.490156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:52.490065 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="815eeede-5d37-4219-8a39-8b122253ced2" path="/var/lib/kubelet/pods/815eeede-5d37-4219-8a39-8b122253ced2/volumes" Oct 30 17:47:56.855100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:56.855062 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" podUID="d293ba72-71a4-4d3d-b559-82f1b009a5ef" containerName="main" containerID="cri-o://3875ae492a795c5f4316acb79b6f0bf9c037440ad672771ac88a2e2a247f25d6" gracePeriod=45 Oct 30 17:47:56.882334 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:56.882302 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:57.026857 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.026821 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:57.129017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.128987 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:57.259154 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.259117 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t"] Oct 30 17:47:57.259624 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.259590 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerName="istio-proxy" containerID="cri-o://9f5e457c5564bab38602e9566e0da68b71650a52c9903b418f5c5a0f26858a52" gracePeriod=30 Oct 30 17:47:57.281128 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.281091 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tls-certs\") pod \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " Oct 30 17:47:57.281330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.281164 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-home\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-home\") pod \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " Oct 30 17:47:57.281330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.281236 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-dshm\") pod \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " Oct 30 17:47:57.281330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.281262 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tmp-dir\") pod \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " Oct 30 17:47:57.281330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.281300 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-model-cache\") pod \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " Oct 30 17:47:57.281330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.281323 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d293ba72-71a4-4d3d-b559-82f1b009a5ef-kube-api-access-nsqwm\" (UniqueName: \"kubernetes.io/projected/d293ba72-71a4-4d3d-b559-82f1b009a5ef-kube-api-access-nsqwm\") pod \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\" (UID: \"d293ba72-71a4-4d3d-b559-82f1b009a5ef\") " Oct 30 17:47:57.281621 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.281482 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-home" pod "d293ba72-71a4-4d3d-b559-82f1b009a5ef" (UID: "d293ba72-71a4-4d3d-b559-82f1b009a5ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:57.281742 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.281697 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tmp-dir" pod "d293ba72-71a4-4d3d-b559-82f1b009a5ef" (UID: "d293ba72-71a4-4d3d-b559-82f1b009a5ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:57.281812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.281734 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-model-cache" pod "d293ba72-71a4-4d3d-b559-82f1b009a5ef" (UID: "d293ba72-71a4-4d3d-b559-82f1b009a5ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:57.283977 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.283945 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d293ba72-71a4-4d3d-b559-82f1b009a5ef-kube-api-access-nsqwm" pod "d293ba72-71a4-4d3d-b559-82f1b009a5ef" (UID: "d293ba72-71a4-4d3d-b559-82f1b009a5ef"). InnerVolumeSpecName "kube-api-access-nsqwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:47:57.284109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.284012 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-dshm" pod "d293ba72-71a4-4d3d-b559-82f1b009a5ef" (UID: "d293ba72-71a4-4d3d-b559-82f1b009a5ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:47:57.284168 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.284134 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tls-certs" pod "d293ba72-71a4-4d3d-b559-82f1b009a5ef" (UID: "d293ba72-71a4-4d3d-b559-82f1b009a5ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:47:57.341079 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.341037 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:47:57.382558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.382431 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:47:57.382558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.382470 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:47:57.382558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.382483 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:47:57.382558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.382495 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsqwm\" (UniqueName: \"kubernetes.io/projected/d293ba72-71a4-4d3d-b559-82f1b009a5ef-kube-api-access-nsqwm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:47:57.382558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.382507 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d293ba72-71a4-4d3d-b559-82f1b009a5ef-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:47:57.382558 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.382518 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d293ba72-71a4-4d3d-b559-82f1b009a5ef-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:47:57.731842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.731747 4828 generic.go:356] "Generic (PLEG): container finished" podID="d293ba72-71a4-4d3d-b559-82f1b009a5ef" containerID="3875ae492a795c5f4316acb79b6f0bf9c037440ad672771ac88a2e2a247f25d6" exitCode=0 Oct 30 17:47:57.731842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.731803 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" Oct 30 17:47:57.731842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.731807 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" event={"ID":"d293ba72-71a4-4d3d-b559-82f1b009a5ef","Type":"ContainerDied","Data":"3875ae492a795c5f4316acb79b6f0bf9c037440ad672771ac88a2e2a247f25d6"} Oct 30 17:47:57.732172 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.731859 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z" event={"ID":"d293ba72-71a4-4d3d-b559-82f1b009a5ef","Type":"ContainerDied","Data":"134653c1cfedceb3903d376f9578f622097ba5ac167b256fc1ccf7bc2e767d75"} Oct 30 17:47:57.732172 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.731882 4828 scope.go:122] "RemoveContainer" containerID="3875ae492a795c5f4316acb79b6f0bf9c037440ad672771ac88a2e2a247f25d6" Oct 30 17:47:57.742953 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.742912 4828 scope.go:122] "RemoveContainer" containerID="3875ae492a795c5f4316acb79b6f0bf9c037440ad672771ac88a2e2a247f25d6" Oct 30 17:47:57.743250 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:47:57.743229 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3875ae492a795c5f4316acb79b6f0bf9c037440ad672771ac88a2e2a247f25d6\": container with ID starting with 3875ae492a795c5f4316acb79b6f0bf9c037440ad672771ac88a2e2a247f25d6 not found: ID does not exist" containerID="3875ae492a795c5f4316acb79b6f0bf9c037440ad672771ac88a2e2a247f25d6" Oct 30 17:47:57.743318 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.743263 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3875ae492a795c5f4316acb79b6f0bf9c037440ad672771ac88a2e2a247f25d6"} err="failed to get container status \"3875ae492a795c5f4316acb79b6f0bf9c037440ad672771ac88a2e2a247f25d6\": rpc error: code = NotFound desc = could not find container \"3875ae492a795c5f4316acb79b6f0bf9c037440ad672771ac88a2e2a247f25d6\": container with ID starting with 3875ae492a795c5f4316acb79b6f0bf9c037440ad672771ac88a2e2a247f25d6 not found: ID does not exist" Oct 30 17:47:57.782433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.782404 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z"] Oct 30 17:47:57.786826 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:57.786797 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-c118c3/simulator-single-main-c118c3-kserve-59df49bf85-v774z"] Oct 30 17:47:58.488446 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:58.488406 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d293ba72-71a4-4d3d-b559-82f1b009a5ef" path="/var/lib/kubelet/pods/d293ba72-71a4-4d3d-b559-82f1b009a5ef/volumes" Oct 30 17:47:59.249640 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:59.249599 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" podUID="6a6d1441-5515-4a66-b9e5-4811f2ff7a6a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:47:59.250093 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:47:59.250068 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" podUID="a913e1db-f066-4ce7-bf78-46fd2f584436" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:48:01.692754 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:01.692713 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:48:02.307485 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.307438 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:48:02.523599 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.523536 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:48:02.637034 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.636999 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-podinfo\") pod \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " Oct 30 17:48:02.637238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.637050 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-data\") pod \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " Oct 30 17:48:02.637238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.637084 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-socket\") pod \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " Oct 30 17:48:02.637238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.637103 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-certs\") pod \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " Oct 30 17:48:02.637238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.637194 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-kube-api-access-nz258\" (UniqueName: \"kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-kube-api-access-nz258\") pod \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " Oct 30 17:48:02.637238 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.637221 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-token\" (UniqueName: \"kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-token\") pod \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " Oct 30 17:48:02.637539 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.637300 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-envoy\") pod \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " Oct 30 17:48:02.637539 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.637346 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istiod-ca-cert\") pod \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " Oct 30 17:48:02.637539 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.637466 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-socket" pod "95d7a6a4-aa78-4aa3-b31d-05682ec67e03" (UID: "95d7a6a4-aa78-4aa3-b31d-05682ec67e03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:02.637539 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.637476 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-data" pod "95d7a6a4-aa78-4aa3-b31d-05682ec67e03" (UID: "95d7a6a4-aa78-4aa3-b31d-05682ec67e03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:02.637539 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.637492 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-certs" pod "95d7a6a4-aa78-4aa3-b31d-05682ec67e03" (UID: "95d7a6a4-aa78-4aa3-b31d-05682ec67e03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:02.637809 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.637570 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-credential-socket\") pod \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\" (UID: \"95d7a6a4-aa78-4aa3-b31d-05682ec67e03\") " Oct 30 17:48:02.637879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.637840 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-credential-socket" pod "95d7a6a4-aa78-4aa3-b31d-05682ec67e03" (UID: "95d7a6a4-aa78-4aa3-b31d-05682ec67e03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:02.637879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.637866 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istiod-ca-cert" pod "95d7a6a4-aa78-4aa3-b31d-05682ec67e03" (UID: "95d7a6a4-aa78-4aa3-b31d-05682ec67e03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:48:02.638370 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.638344 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:02.638473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.638374 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:02.638473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.638389 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:02.638473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.638403 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:02.638473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.638418 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:02.639584 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.639558 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-envoy" pod "95d7a6a4-aa78-4aa3-b31d-05682ec67e03" (UID: "95d7a6a4-aa78-4aa3-b31d-05682ec67e03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:02.639747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.639731 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-podinfo" pod "95d7a6a4-aa78-4aa3-b31d-05682ec67e03" (UID: "95d7a6a4-aa78-4aa3-b31d-05682ec67e03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:48:02.640036 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.640021 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-kube-api-access-nz258" pod "95d7a6a4-aa78-4aa3-b31d-05682ec67e03" (UID: "95d7a6a4-aa78-4aa3-b31d-05682ec67e03"). InnerVolumeSpecName "kube-api-access-nz258". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:48:02.640251 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.640237 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-token" pod "95d7a6a4-aa78-4aa3-b31d-05682ec67e03" (UID: "95d7a6a4-aa78-4aa3-b31d-05682ec67e03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:48:02.739191 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.739151 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz258\" (UniqueName: \"kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-kube-api-access-nz258\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:02.739191 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.739186 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:02.739191 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.739196 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:02.739653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.739204 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95d7a6a4-aa78-4aa3-b31d-05682ec67e03-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:02.771462 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.771423 4828 generic.go:356] "Generic (PLEG): container finished" podID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerID="9f5e457c5564bab38602e9566e0da68b71650a52c9903b418f5c5a0f26858a52" exitCode=0 Oct 30 17:48:02.771639 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.771480 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" Oct 30 17:48:02.771639 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.771509 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" event={"ID":"95d7a6a4-aa78-4aa3-b31d-05682ec67e03","Type":"ContainerDied","Data":"9f5e457c5564bab38602e9566e0da68b71650a52c9903b418f5c5a0f26858a52"} Oct 30 17:48:02.771639 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.771585 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t" event={"ID":"95d7a6a4-aa78-4aa3-b31d-05682ec67e03","Type":"ContainerDied","Data":"2b0bd2af97963ac0fc8c587034d5a19414b2a1ae857eb05044ff77635de5843a"} Oct 30 17:48:02.771639 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.771608 4828 scope.go:122] "RemoveContainer" containerID="9f5e457c5564bab38602e9566e0da68b71650a52c9903b418f5c5a0f26858a52" Oct 30 17:48:02.781344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.781327 4828 scope.go:122] "RemoveContainer" containerID="200fd1bb16fb79d18632e33f67f870cf880512356cf6f1826657bfdea62b5178" Oct 30 17:48:02.790135 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.790113 4828 scope.go:122] "RemoveContainer" containerID="9f5e457c5564bab38602e9566e0da68b71650a52c9903b418f5c5a0f26858a52" Oct 30 17:48:02.790405 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:48:02.790383 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5e457c5564bab38602e9566e0da68b71650a52c9903b418f5c5a0f26858a52\": container with ID starting with 9f5e457c5564bab38602e9566e0da68b71650a52c9903b418f5c5a0f26858a52 not found: ID does not exist" containerID="9f5e457c5564bab38602e9566e0da68b71650a52c9903b418f5c5a0f26858a52" Oct 30 17:48:02.790463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.790414 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5e457c5564bab38602e9566e0da68b71650a52c9903b418f5c5a0f26858a52"} err="failed to get container status \"9f5e457c5564bab38602e9566e0da68b71650a52c9903b418f5c5a0f26858a52\": rpc error: code = NotFound desc = could not find container \"9f5e457c5564bab38602e9566e0da68b71650a52c9903b418f5c5a0f26858a52\": container with ID starting with 9f5e457c5564bab38602e9566e0da68b71650a52c9903b418f5c5a0f26858a52 not found: ID does not exist" Oct 30 17:48:02.790463 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.790434 4828 scope.go:122] "RemoveContainer" containerID="200fd1bb16fb79d18632e33f67f870cf880512356cf6f1826657bfdea62b5178" Oct 30 17:48:02.790723 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:48:02.790703 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200fd1bb16fb79d18632e33f67f870cf880512356cf6f1826657bfdea62b5178\": container with ID starting with 200fd1bb16fb79d18632e33f67f870cf880512356cf6f1826657bfdea62b5178 not found: ID does not exist" containerID="200fd1bb16fb79d18632e33f67f870cf880512356cf6f1826657bfdea62b5178" Oct 30 17:48:02.790761 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.790730 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200fd1bb16fb79d18632e33f67f870cf880512356cf6f1826657bfdea62b5178"} err="failed to get container status \"200fd1bb16fb79d18632e33f67f870cf880512356cf6f1826657bfdea62b5178\": rpc error: code = NotFound desc = could not find container \"200fd1bb16fb79d18632e33f67f870cf880512356cf6f1826657bfdea62b5178\": container with ID starting with 200fd1bb16fb79d18632e33f67f870cf880512356cf6f1826657bfdea62b5178 not found: ID does not exist" Oct 30 17:48:02.824305 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.824268 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t"] Oct 30 17:48:02.829775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:02.829751 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-c118c3-openshift-default-6cd6b955b529k6t"] Oct 30 17:48:04.489185 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:04.489154 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" path="/var/lib/kubelet/pods/95d7a6a4-aa78-4aa3-b31d-05682ec67e03/volumes" Oct 30 17:48:07.911102 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:48:07.911058 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:48:08.459330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:08.459290 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:48:09.249778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:09.249727 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" podUID="a913e1db-f066-4ce7-bf78-46fd2f584436" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:48:09.250301 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:09.249744 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" podUID="6a6d1441-5515-4a66-b9e5-4811f2ff7a6a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:48:09.968138 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:09.968095 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:48:12.889243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:12.889206 4828 generic.go:356] "Generic (PLEG): container finished" podID="a913e1db-f066-4ce7-bf78-46fd2f584436" containerID="c8eb291ab508a724e091e73048852f0fa17220b0dba354af7cab8f16bcdac59d" exitCode=0 Oct 30 17:48:12.898302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:12.898178 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" event={"ID":"a913e1db-f066-4ce7-bf78-46fd2f584436","Type":"ContainerDied","Data":"c8eb291ab508a724e091e73048852f0fa17220b0dba354af7cab8f16bcdac59d"} Oct 30 17:48:13.007686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.007647 4828 generic.go:356] "Generic (PLEG): container finished" podID="6a6d1441-5515-4a66-b9e5-4811f2ff7a6a" containerID="a6586b22ecbd35ad47786d96ab83885ad6024fb833caa9ffa911d96234f5df9e" exitCode=0 Oct 30 17:48:13.008016 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.007989 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" event={"ID":"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a","Type":"ContainerDied","Data":"a6586b22ecbd35ad47786d96ab83885ad6024fb833caa9ffa911d96234f5df9e"} Oct 30 17:48:13.020956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.018220 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" Oct 30 17:48:13.175978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.175190 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-kube-api-access-qgpvp\" (UniqueName: \"kubernetes.io/projected/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-kube-api-access-qgpvp\") pod \"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a\" (UID: \"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a\") " Oct 30 17:48:13.175978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.175256 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-plugins-config-volume\") pod \"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a\" (UID: \"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a\") " Oct 30 17:48:13.292511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.281697 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-kube-api-access-qgpvp" pod "6a6d1441-5515-4a66-b9e5-4811f2ff7a6a" (UID: "6a6d1441-5515-4a66-b9e5-4811f2ff7a6a"). InnerVolumeSpecName "kube-api-access-qgpvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:48:13.329924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.293494 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-plugins-config-volume" pod "6a6d1441-5515-4a66-b9e5-4811f2ff7a6a" (UID: "6a6d1441-5515-4a66-b9e5-4811f2ff7a6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:48:13.389603 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.387676 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgpvp\" (UniqueName: \"kubernetes.io/projected/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-kube-api-access-qgpvp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:13.389603 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.387718 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:13.539925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.520663 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" Oct 30 17:48:13.589992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.589425 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a913e1db-f066-4ce7-bf78-46fd2f584436-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a913e1db-f066-4ce7-bf78-46fd2f584436-plugins-config-volume\") pod \"a913e1db-f066-4ce7-bf78-46fd2f584436\" (UID: \"a913e1db-f066-4ce7-bf78-46fd2f584436\") " Oct 30 17:48:13.590348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.590301 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a913e1db-f066-4ce7-bf78-46fd2f584436-kube-api-access-9wwnp\" (UniqueName: \"kubernetes.io/projected/a913e1db-f066-4ce7-bf78-46fd2f584436-kube-api-access-9wwnp\") pod \"a913e1db-f066-4ce7-bf78-46fd2f584436\" (UID: \"a913e1db-f066-4ce7-bf78-46fd2f584436\") " Oct 30 17:48:13.591838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.589875 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a913e1db-f066-4ce7-bf78-46fd2f584436-plugins-config-volume" pod "a913e1db-f066-4ce7-bf78-46fd2f584436" (UID: "a913e1db-f066-4ce7-bf78-46fd2f584436"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:48:13.595436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.595411 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a913e1db-f066-4ce7-bf78-46fd2f584436-kube-api-access-9wwnp" pod "a913e1db-f066-4ce7-bf78-46fd2f584436" (UID: "a913e1db-f066-4ce7-bf78-46fd2f584436"). InnerVolumeSpecName "kube-api-access-9wwnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:48:13.694170 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.694132 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a913e1db-f066-4ce7-bf78-46fd2f584436-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:13.694433 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:13.694412 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wwnp\" (UniqueName: \"kubernetes.io/projected/a913e1db-f066-4ce7-bf78-46fd2f584436-kube-api-access-9wwnp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:14.015650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:14.015618 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" Oct 30 17:48:14.016176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:14.015662 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99" event={"ID":"6a6d1441-5515-4a66-b9e5-4811f2ff7a6a","Type":"ContainerDied","Data":"329a35ebe82f69e466b0ae405e6d114dfa76bdd112baa6e7509a5eadc2640a83"} Oct 30 17:48:14.016176 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:14.015734 4828 scope.go:122] "RemoveContainer" containerID="a6586b22ecbd35ad47786d96ab83885ad6024fb833caa9ffa911d96234f5df9e" Oct 30 17:48:14.017298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:14.017275 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" Oct 30 17:48:14.017422 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:14.017275 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g" event={"ID":"a913e1db-f066-4ce7-bf78-46fd2f584436","Type":"ContainerDied","Data":"9f5b87e038c13cf4b3ca9b0a66336244cecfe10803056a0a7e7af1db4893de89"} Oct 30 17:48:14.026052 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:14.026032 4828 scope.go:122] "RemoveContainer" containerID="c8eb291ab508a724e091e73048852f0fa17220b0dba354af7cab8f16bcdac59d" Oct 30 17:48:14.066313 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:14.066280 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99"] Oct 30 17:48:14.077542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:14.077510 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-c118c3-7fb97b945d8k99"] Oct 30 17:48:14.103325 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:14.103281 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g"] Oct 30 17:48:14.110460 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:14.110420 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-c118c3-67d6cz7c2g"] Oct 30 17:48:14.490104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:14.490021 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a6d1441-5515-4a66-b9e5-4811f2ff7a6a" path="/var/lib/kubelet/pods/6a6d1441-5515-4a66-b9e5-4811f2ff7a6a/volumes" Oct 30 17:48:14.490617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:14.490599 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a913e1db-f066-4ce7-bf78-46fd2f584436" path="/var/lib/kubelet/pods/a913e1db-f066-4ce7-bf78-46fd2f584436/volumes" Oct 30 17:48:18.390746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:18.390699 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:48:20.187600 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:20.187560 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:48:26.108512 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:26.108476 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj"] Oct 30 17:48:26.108983 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:26.108676 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" podUID="987bce29-ba9c-4a66-b285-f8706441e626" containerName="manager" containerID="cri-o://0a46fc60812b679ae1e88c81173c005e6f0da49df6a530055360c5958661fb8f" gracePeriod=10 Oct 30 17:48:26.157450 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:26.157411 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:48:26.373862 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:26.373797 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" Oct 30 17:48:26.443735 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:26.443697 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987bce29-ba9c-4a66-b285-f8706441e626-kube-api-access-4jr9c\" (UniqueName: \"kubernetes.io/projected/987bce29-ba9c-4a66-b285-f8706441e626-kube-api-access-4jr9c\") pod \"987bce29-ba9c-4a66-b285-f8706441e626\" (UID: \"987bce29-ba9c-4a66-b285-f8706441e626\") " Oct 30 17:48:26.443926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:26.443827 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987bce29-ba9c-4a66-b285-f8706441e626-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/987bce29-ba9c-4a66-b285-f8706441e626-extensions-socket-volume\") pod \"987bce29-ba9c-4a66-b285-f8706441e626\" (UID: \"987bce29-ba9c-4a66-b285-f8706441e626\") " Oct 30 17:48:26.444254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:26.444229 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987bce29-ba9c-4a66-b285-f8706441e626-extensions-socket-volume" pod "987bce29-ba9c-4a66-b285-f8706441e626" (UID: "987bce29-ba9c-4a66-b285-f8706441e626"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:48:26.446004 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:26.445966 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987bce29-ba9c-4a66-b285-f8706441e626-kube-api-access-4jr9c" pod "987bce29-ba9c-4a66-b285-f8706441e626" (UID: "987bce29-ba9c-4a66-b285-f8706441e626"). InnerVolumeSpecName "kube-api-access-4jr9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:48:26.545869 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:26.545834 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jr9c\" (UniqueName: \"kubernetes.io/projected/987bce29-ba9c-4a66-b285-f8706441e626-kube-api-access-4jr9c\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:26.545869 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:26.545866 4828 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/987bce29-ba9c-4a66-b285-f8706441e626-extensions-socket-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:48:27.092937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:27.092881 4828 generic.go:356] "Generic (PLEG): container finished" podID="987bce29-ba9c-4a66-b285-f8706441e626" containerID="0a46fc60812b679ae1e88c81173c005e6f0da49df6a530055360c5958661fb8f" exitCode=0 Oct 30 17:48:27.093134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:27.092947 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" Oct 30 17:48:27.093134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:27.092954 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" event={"ID":"987bce29-ba9c-4a66-b285-f8706441e626","Type":"ContainerDied","Data":"0a46fc60812b679ae1e88c81173c005e6f0da49df6a530055360c5958661fb8f"} Oct 30 17:48:27.093134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:27.092996 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj" event={"ID":"987bce29-ba9c-4a66-b285-f8706441e626","Type":"ContainerDied","Data":"c73db37b580bdf7ab8d4b09a9e810ac6d97cad630441f8eba423a52ec1bfa7e5"} Oct 30 17:48:27.093134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:27.093016 4828 scope.go:122] "RemoveContainer" containerID="0a46fc60812b679ae1e88c81173c005e6f0da49df6a530055360c5958661fb8f" Oct 30 17:48:27.103012 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:27.102989 4828 scope.go:122] "RemoveContainer" containerID="0a46fc60812b679ae1e88c81173c005e6f0da49df6a530055360c5958661fb8f" Oct 30 17:48:27.103315 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:48:27.103288 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a46fc60812b679ae1e88c81173c005e6f0da49df6a530055360c5958661fb8f\": container with ID starting with 0a46fc60812b679ae1e88c81173c005e6f0da49df6a530055360c5958661fb8f not found: ID does not exist" containerID="0a46fc60812b679ae1e88c81173c005e6f0da49df6a530055360c5958661fb8f" Oct 30 17:48:27.103428 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:27.103329 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a46fc60812b679ae1e88c81173c005e6f0da49df6a530055360c5958661fb8f"} err="failed to get container status \"0a46fc60812b679ae1e88c81173c005e6f0da49df6a530055360c5958661fb8f\": rpc error: code = NotFound desc = could not find container \"0a46fc60812b679ae1e88c81173c005e6f0da49df6a530055360c5958661fb8f\": container with ID starting with 0a46fc60812b679ae1e88c81173c005e6f0da49df6a530055360c5958661fb8f not found: ID does not exist" Oct 30 17:48:27.156301 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:27.156264 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj"] Oct 30 17:48:27.169625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:27.169574 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-n2gnj"] Oct 30 17:48:28.488581 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:48:28.488547 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987bce29-ba9c-4a66-b285-f8706441e626" path="/var/lib/kubelet/pods/987bce29-ba9c-4a66-b285-f8706441e626/volumes" Oct 30 17:48:37.910419 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:48:37.910371 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:49:01.820409 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:01.820372 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:49:02.433214 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:02.433177 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:49:07.911161 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:49:07.911121 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:49:10.740964 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:10.740923 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:49:29.294302 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.294207 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr"] Oct 30 17:49:29.294832 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.294812 4828 cpu_manager.go:414] "containerMap: removing container" podUID="815eeede-5d37-4219-8a39-8b122253ced2" containerName="maas-api" Oct 30 17:49:29.294937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.294834 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815eeede-5d37-4219-8a39-8b122253ced2" containerName="maas-api" Oct 30 17:49:29.294937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.294860 4828 cpu_manager.go:414] "containerMap: removing container" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerName="istio-proxy" Oct 30 17:49:29.294937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.294868 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerName="istio-proxy" Oct 30 17:49:29.294937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.294879 4828 cpu_manager.go:414] "containerMap: removing container" podUID="6a6d1441-5515-4a66-b9e5-4811f2ff7a6a" containerName="payload-processing" Oct 30 17:49:29.294937 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.294887 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a6d1441-5515-4a66-b9e5-4811f2ff7a6a" containerName="payload-processing" Oct 30 17:49:29.295231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.294945 4828 cpu_manager.go:414] "containerMap: removing container" podUID="d293ba72-71a4-4d3d-b559-82f1b009a5ef" containerName="main" Oct 30 17:49:29.295231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.294954 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d293ba72-71a4-4d3d-b559-82f1b009a5ef" containerName="main" Oct 30 17:49:29.295231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.294989 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a913e1db-f066-4ce7-bf78-46fd2f584436" containerName="payload-pre-processing" Oct 30 17:49:29.295231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.294997 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a913e1db-f066-4ce7-bf78-46fd2f584436" containerName="payload-pre-processing" Oct 30 17:49:29.295231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.295011 4828 cpu_manager.go:414] "containerMap: removing container" podUID="987bce29-ba9c-4a66-b285-f8706441e626" containerName="manager" Oct 30 17:49:29.295231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.295018 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987bce29-ba9c-4a66-b285-f8706441e626" containerName="manager" Oct 30 17:49:29.295231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.295133 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="d293ba72-71a4-4d3d-b559-82f1b009a5ef" containerName="main" Oct 30 17:49:29.295231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.295150 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a913e1db-f066-4ce7-bf78-46fd2f584436" containerName="payload-pre-processing" Oct 30 17:49:29.295231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.295161 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerName="istio-proxy" Oct 30 17:49:29.295231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.295171 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="987bce29-ba9c-4a66-b285-f8706441e626" containerName="manager" Oct 30 17:49:29.295231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.295184 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="815eeede-5d37-4219-8a39-8b122253ced2" containerName="maas-api" Oct 30 17:49:29.295231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.295195 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerName="istio-proxy" Oct 30 17:49:29.295231 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.295204 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a6d1441-5515-4a66-b9e5-4811f2ff7a6a" containerName="payload-processing" Oct 30 17:49:29.300056 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.300035 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr" Oct 30 17:49:29.316001 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.315970 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr"] Oct 30 17:49:29.324468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.324437 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jdb28\"" Oct 30 17:49:29.358711 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.358671 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xrs7\" (UniqueName: \"kubernetes.io/projected/175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3-kube-api-access-4xrs7\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mrcfr\" (UID: \"175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr" Oct 30 17:49:29.358887 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.358777 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mrcfr\" (UID: \"175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr" Oct 30 17:49:29.460003 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.459955 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrs7\" (UniqueName: \"kubernetes.io/projected/175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3-kube-api-access-4xrs7\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mrcfr\" (UID: \"175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr" Oct 30 17:49:29.460223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.460059 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mrcfr\" (UID: \"175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr" Oct 30 17:49:29.460485 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.460463 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mrcfr\" (UID: \"175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr" Oct 30 17:49:29.483332 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.483300 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrs7\" (UniqueName: \"kubernetes.io/projected/175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3-kube-api-access-4xrs7\") pod \"kuadrant-operator-controller-manager-57bb9bfff-mrcfr\" (UID: \"175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr" Oct 30 17:49:29.613858 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.613769 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr" Oct 30 17:49:29.949968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:29.949865 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr"] Oct 30 17:49:29.965945 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:49:29.965911 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175fb4ce_bdfb_4d1d_bd67_0b59fe27f8d3.slice/crio-ea21492f0b506b7c1da0d9fbd2df077b2208e1b1da133109604870eb2cac8246 WatchSource:0}: Error finding container ea21492f0b506b7c1da0d9fbd2df077b2208e1b1da133109604870eb2cac8246: Status 404 returned error can't find the container with id ea21492f0b506b7c1da0d9fbd2df077b2208e1b1da133109604870eb2cac8246 Oct 30 17:49:30.447414 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:30.447376 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr" event={"ID":"175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3","Type":"ContainerStarted","Data":"8b4a6a61906fb4a7eacbe162c96638d3b7d57a78525054b2b02e4a17b57da0cf"} Oct 30 17:49:30.447818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:30.447436 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr" event={"ID":"175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3","Type":"ContainerStarted","Data":"ea21492f0b506b7c1da0d9fbd2df077b2208e1b1da133109604870eb2cac8246"} Oct 30 17:49:30.447818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:30.447472 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr" Oct 30 17:49:30.517456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:30.517393 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr" podStartSLOduration=1.517374837 podStartE2EDuration="1.517374837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:49:30.504140275 +0000 UTC m=+3748.149650455" watchObservedRunningTime="2026-10-30 17:49:30.517374837 +0000 UTC m=+3748.162885018" Oct 30 17:49:37.910465 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:49:37.910422 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:49:41.455974 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:41.455936 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-mrcfr" Oct 30 17:49:43.595436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:43.595399 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:49:43.795462 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:43.795420 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:49:52.401924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:52.401027 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r26lb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:49:52.401924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:49:52.401125 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" podUID="1fc9a9e7-416b-4b87-912e-a524f41e0d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:50:01.926331 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:01.926295 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:02.549390 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:02.549346 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:07.924031 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:50:07.923977 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:50:22.531295 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:22.531256 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:22.909935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:22.909821 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb"] Oct 30 17:50:22.910929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:22.910521 4828 cpu_manager.go:414] "containerMap: removing container" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerName="istio-proxy" Oct 30 17:50:22.910929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:22.910549 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d7a6a4-aa78-4aa3-b31d-05682ec67e03" containerName="istio-proxy" Oct 30 17:50:22.914225 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:22.914200 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:22.928810 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:22.928777 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ce3c55-openshift-default-dockercfg-vm5cp\"" Oct 30 17:50:22.943298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:22.943263 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb"] Oct 30 17:50:23.006173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.006122 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:23.075548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.075507 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-data\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.075755 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.075549 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-credential-socket\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.075839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.075750 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvf7d\" (UniqueName: \"kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-kube-api-access-bvf7d\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.075839 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.075812 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-podinfo\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.075988 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.075963 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-socket\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.076038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.075990 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-token\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.076038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.076024 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-envoy\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.076139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.076129 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istiod-ca-cert\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.076179 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.076148 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-certs\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.176871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.176771 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvf7d\" (UniqueName: \"kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-kube-api-access-bvf7d\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.176871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.176824 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-podinfo\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.176871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.176856 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-socket\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.176871 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.176874 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-token\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.177279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.176931 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-envoy\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.177279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.176985 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istiod-ca-cert\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.177279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.177005 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-certs\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.177279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.177046 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-data\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.177279 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.177203 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-credential-socket\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.177566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.177326 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-socket\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.177566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.177365 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-data\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.177566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.177498 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-certs\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.177566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.177530 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-credential-socket\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.177986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.177959 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istiod-ca-cert\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.179641 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.179618 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-envoy\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.180233 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.180209 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-podinfo\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.185209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.185181 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:23.200123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.200093 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-token\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.200364 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.200338 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvf7d\" (UniqueName: \"kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-kube-api-access-bvf7d\") pod \"e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.229356 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.229317 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:23.554149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.554114 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb"] Oct 30 17:50:23.556572 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:50:23.556537 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c0b5e6_73ab_4398_a200_c7ef5e974373.slice/crio-037e7b9e7cf13419bbddb3186148575db10c6e458da881c8d01bc72a8b904ec6 WatchSource:0}: Error finding container 037e7b9e7cf13419bbddb3186148575db10c6e458da881c8d01bc72a8b904ec6: Status 404 returned error can't find the container with id 037e7b9e7cf13419bbddb3186148575db10c6e458da881c8d01bc72a8b904ec6 Oct 30 17:50:23.558737 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.558707 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:50:23.558814 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.558770 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:50:23.558814 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.558797 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:50:23.832684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.832574 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" event={"ID":"c5c0b5e6-73ab-4398-a200-c7ef5e974373","Type":"ContainerStarted","Data":"06bdfc415363bdfd93c272d903eeb10f4e3dae4aab670ceec963a78408bcf7bf"} Oct 30 17:50:23.832684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.832649 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" event={"ID":"c5c0b5e6-73ab-4398-a200-c7ef5e974373","Type":"ContainerStarted","Data":"037e7b9e7cf13419bbddb3186148575db10c6e458da881c8d01bc72a8b904ec6"} Oct 30 17:50:23.877807 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:23.877742 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" podStartSLOduration=1.877728239 podStartE2EDuration="1.877728239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:50:23.877156606 +0000 UTC m=+3801.522666781" watchObservedRunningTime="2026-10-30 17:50:23.877728239 +0000 UTC m=+3801.523238419" Oct 30 17:50:24.229737 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:24.229644 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:24.234847 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:24.234818 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:24.838706 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:24.838663 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:24.839645 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:24.839622 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:26.450283 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:26.450242 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:26.608133 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:26.608089 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:28.007120 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.007085 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k"] Oct 30 17:50:28.010822 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.010801 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.020276 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.020248 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlwf2\" (UniqueName: \"kubernetes.io/projected/ed6b5197-637f-475c-9c75-4a9277b37746-kube-api-access-vlwf2\") pod \"maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.020412 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.020359 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6b5197-637f-475c-9c75-4a9277b37746-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.020412 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.020391 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.020552 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.020428 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.031291 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.031258 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ce3c55-serving-cert\"" Oct 30 17:50:28.041577 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.041541 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k"] Oct 30 17:50:28.121920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.121865 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlwf2\" (UniqueName: \"kubernetes.io/projected/ed6b5197-637f-475c-9c75-4a9277b37746-kube-api-access-vlwf2\") pod \"maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.122129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.121993 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6b5197-637f-475c-9c75-4a9277b37746-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.122129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.122028 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.122129 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.122067 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.122427 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:50:28.122404 4828 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-serving-cert: secret "maas-api-e2e-worker-single-main-ce3c55-serving-cert" not found Oct 30 17:50:28.122617 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:50:28.122602 4828 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-maas-api-tls podName:ed6b5197-637f-475c-9c75-4a9277b37746 nodeName:}" failed. No retries permitted until 2026-10-30 17:50:28.622579583 +0000 UTC m=+3806.268089763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-maas-api-tls") pod "maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" (UID: "ed6b5197-637f-475c-9c75-4a9277b37746") : secret "maas-api-e2e-worker-single-main-ce3c55-serving-cert" not found Oct 30 17:50:28.123948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.123917 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6b5197-637f-475c-9c75-4a9277b37746-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.125572 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.125540 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.143795 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.143759 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:28.148105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.148076 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlwf2\" (UniqueName: \"kubernetes.io/projected/ed6b5197-637f-475c-9c75-4a9277b37746-kube-api-access-vlwf2\") pod \"maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.286805 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.286765 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p"] Oct 30 17:50:28.290673 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.290649 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" Oct 30 17:50:28.308309 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.308278 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-1b5a3cb5-dockercfg-svvxs\"" Oct 30 17:50:28.311202 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.311175 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ce3c55\"" Oct 30 17:50:28.311344 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.311216 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p"] Oct 30 17:50:28.318297 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.318262 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:28.323544 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.323512 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1e4365-35cf-49ed-b291-4fa18b1411a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p\" (UID: \"1b1e4365-35cf-49ed-b291-4fa18b1411a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" Oct 30 17:50:28.323695 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.323636 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52kcp\" (UniqueName: \"kubernetes.io/projected/1b1e4365-35cf-49ed-b291-4fa18b1411a4-kube-api-access-52kcp\") pod \"payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p\" (UID: \"1b1e4365-35cf-49ed-b291-4fa18b1411a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" Oct 30 17:50:28.375473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.375430 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4"] Oct 30 17:50:28.379294 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.379266 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" Oct 30 17:50:28.434533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.429718 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4"] Oct 30 17:50:28.437530 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.437486 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1e4365-35cf-49ed-b291-4fa18b1411a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p\" (UID: \"1b1e4365-35cf-49ed-b291-4fa18b1411a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" Oct 30 17:50:28.437729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.437627 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52kcp\" (UniqueName: \"kubernetes.io/projected/1b1e4365-35cf-49ed-b291-4fa18b1411a4-kube-api-access-52kcp\") pod \"payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p\" (UID: \"1b1e4365-35cf-49ed-b291-4fa18b1411a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" Oct 30 17:50:28.438110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.437785 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjrr\" (UniqueName: \"kubernetes.io/projected/89d803d3-631d-430b-8a57-3ed320ee6382-kube-api-access-sfjrr\") pod \"payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4\" (UID: \"89d803d3-631d-430b-8a57-3ed320ee6382\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" Oct 30 17:50:28.438110 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.437913 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d803d3-631d-430b-8a57-3ed320ee6382-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4\" (UID: \"89d803d3-631d-430b-8a57-3ed320ee6382\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" Oct 30 17:50:28.438457 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.438426 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1e4365-35cf-49ed-b291-4fa18b1411a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p\" (UID: \"1b1e4365-35cf-49ed-b291-4fa18b1411a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" Oct 30 17:50:28.462780 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.462743 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52kcp\" (UniqueName: \"kubernetes.io/projected/1b1e4365-35cf-49ed-b291-4fa18b1411a4-kube-api-access-52kcp\") pod \"payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p\" (UID: \"1b1e4365-35cf-49ed-b291-4fa18b1411a4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" Oct 30 17:50:28.469432 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.469392 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:28.538561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.538469 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjrr\" (UniqueName: \"kubernetes.io/projected/89d803d3-631d-430b-8a57-3ed320ee6382-kube-api-access-sfjrr\") pod \"payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4\" (UID: \"89d803d3-631d-430b-8a57-3ed320ee6382\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" Oct 30 17:50:28.538561 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.538548 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d803d3-631d-430b-8a57-3ed320ee6382-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4\" (UID: \"89d803d3-631d-430b-8a57-3ed320ee6382\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" Oct 30 17:50:28.539333 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.539311 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d803d3-631d-430b-8a57-3ed320ee6382-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4\" (UID: \"89d803d3-631d-430b-8a57-3ed320ee6382\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" Oct 30 17:50:28.563662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.563625 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjrr\" (UniqueName: \"kubernetes.io/projected/89d803d3-631d-430b-8a57-3ed320ee6382-kube-api-access-sfjrr\") pod \"payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4\" (UID: \"89d803d3-631d-430b-8a57-3ed320ee6382\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" Oct 30 17:50:28.605503 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.605470 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" Oct 30 17:50:28.640247 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.640206 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.642835 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.642805 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:28.695609 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.695573 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" Oct 30 17:50:28.916167 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.915007 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p"] Oct 30 17:50:28.918230 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:50:28.918192 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1e4365_35cf_49ed_b291_4fa18b1411a4.slice/crio-acfbba415d91e0ff3735e8993a4b917648d12f6709e4fc697b517094ccef31e2 WatchSource:0}: Error finding container acfbba415d91e0ff3735e8993a4b917648d12f6709e4fc697b517094ccef31e2: Status 404 returned error can't find the container with id acfbba415d91e0ff3735e8993a4b917648d12f6709e4fc697b517094ccef31e2 Oct 30 17:50:28.924262 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:28.924227 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:29.007022 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.006991 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4"] Oct 30 17:50:29.010214 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:50:29.010180 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d803d3_631d_430b_8a57_3ed320ee6382.slice/crio-cb482d32ab2bd0ed64c1bbddfd11bf964b67cdd88482d2001a44e8ea1cf4764b WatchSource:0}: Error finding container cb482d32ab2bd0ed64c1bbddfd11bf964b67cdd88482d2001a44e8ea1cf4764b: Status 404 returned error can't find the container with id cb482d32ab2bd0ed64c1bbddfd11bf964b67cdd88482d2001a44e8ea1cf4764b Oct 30 17:50:29.237183 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.237152 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k"] Oct 30 17:50:29.239189 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:50:29.239164 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6b5197_637f_475c_9c75_4a9277b37746.slice/crio-ac884117b1befd1a32d89eb11466344b013f7ab65a6eecab09ee668bba8a0ad5 WatchSource:0}: Error finding container ac884117b1befd1a32d89eb11466344b013f7ab65a6eecab09ee668bba8a0ad5: Status 404 returned error can't find the container with id ac884117b1befd1a32d89eb11466344b013f7ab65a6eecab09ee668bba8a0ad5 Oct 30 17:50:29.868574 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.868535 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" event={"ID":"89d803d3-631d-430b-8a57-3ed320ee6382","Type":"ContainerStarted","Data":"3a6a76f4d85a4b56029a6022937700ec48966c9e5de88e46458c2b7f9b2bfa2a"} Oct 30 17:50:29.868574 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.868590 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" event={"ID":"89d803d3-631d-430b-8a57-3ed320ee6382","Type":"ContainerStarted","Data":"cb482d32ab2bd0ed64c1bbddfd11bf964b67cdd88482d2001a44e8ea1cf4764b"} Oct 30 17:50:29.868845 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.868613 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" Oct 30 17:50:29.870303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.870265 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" event={"ID":"ed6b5197-637f-475c-9c75-4a9277b37746","Type":"ContainerStarted","Data":"515fcb92652aa7b0e2342344016bb0135e70c0402c32d17cdb04ee5d261111dd"} Oct 30 17:50:29.870456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.870322 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" event={"ID":"ed6b5197-637f-475c-9c75-4a9277b37746","Type":"ContainerStarted","Data":"ac884117b1befd1a32d89eb11466344b013f7ab65a6eecab09ee668bba8a0ad5"} Oct 30 17:50:29.870456 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.870352 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:29.871819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.871797 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" event={"ID":"1b1e4365-35cf-49ed-b291-4fa18b1411a4","Type":"ContainerStarted","Data":"50e59345235c79f990733eb7705e7d406a59f854ccb54e9fc4d6e8e89f04a779"} Oct 30 17:50:29.871965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.871832 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" event={"ID":"1b1e4365-35cf-49ed-b291-4fa18b1411a4","Type":"ContainerStarted","Data":"acfbba415d91e0ff3735e8993a4b917648d12f6709e4fc697b517094ccef31e2"} Oct 30 17:50:29.871965 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.871860 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" Oct 30 17:50:29.909368 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.909257 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" podStartSLOduration=1.9092378079999999 podStartE2EDuration="1.909237808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:50:29.908117402 +0000 UTC m=+3807.553627592" watchObservedRunningTime="2026-10-30 17:50:29.909237808 +0000 UTC m=+3807.554747989" Oct 30 17:50:29.947509 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.947442 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" podStartSLOduration=1.947423034 podStartE2EDuration="1.947423034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:50:29.946932073 +0000 UTC m=+3807.592442254" watchObservedRunningTime="2026-10-30 17:50:29.947423034 +0000 UTC m=+3807.592933216" Oct 30 17:50:29.964506 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.964467 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:29.990009 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:29.989948 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" podStartSLOduration=2.57097327 podStartE2EDuration="2.989930772s" totalImagesPullingTime="418.957502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:50:29.988131033 +0000 UTC m=+3807.633641214" watchObservedRunningTime="2026-10-30 17:50:29.989930772 +0000 UTC m=+3807.635440942" Oct 30 17:50:31.823480 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:31.823386 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:33.643571 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:33.643537 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:35.397730 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:35.397693 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:35.894985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:35.894951 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:50:37.156789 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:37.156751 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:37.917478 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:50:37.917444 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:50:39.009676 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:39.009630 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:40.670886 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:40.670846 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:40.881047 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:40.881016 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" Oct 30 17:50:40.881271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:40.881053 4828 patch_prober.go:29] interesting pod/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:50:40.881271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:40.881125 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:50:40.881271 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:40.881213 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" Oct 30 17:50:41.268707 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:41.268670 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:41.923050 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:41.923006 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:41.954248 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:41.954216 4828 generic.go:356] "Generic (PLEG): container finished" podID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerID="06bdfc415363bdfd93c272d903eeb10f4e3dae4aab670ceec963a78408bcf7bf" exitCode=0 Oct 30 17:50:41.954455 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:41.954288 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" event={"ID":"c5c0b5e6-73ab-4398-a200-c7ef5e974373","Type":"ContainerDied","Data":"06bdfc415363bdfd93c272d903eeb10f4e3dae4aab670ceec963a78408bcf7bf"} Oct 30 17:50:41.954735 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:41.954708 4828 scope.go:122] "RemoveContainer" containerID="06bdfc415363bdfd93c272d903eeb10f4e3dae4aab670ceec963a78408bcf7bf" Oct 30 17:50:41.955970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:41.955943 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:50:41.956038 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:41.956013 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:50:41.956080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:41.956045 4828 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 17:50:42.075842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:42.075790 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:42.237289 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:42.237199 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:42.403376 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:42.403339 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:42.962299 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:42.962259 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" event={"ID":"c5c0b5e6-73ab-4398-a200-c7ef5e974373","Type":"ContainerStarted","Data":"7251241bb2f4dd45669a2077a55ee1d41b9fb1ed531499aba794eeb8e68f374e"} Oct 30 17:50:43.230332 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:43.230249 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:43.230519 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:43.230476 4828 patch_prober.go:29] interesting pod/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 17:50:43.230603 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:43.230533 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 17:50:43.994017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:43.993975 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d"] Oct 30 17:50:44.003929 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.003146 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.012666 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.012618 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d"] Oct 30 17:50:44.013265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.013236 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp4tf\" (UniqueName: \"kubernetes.io/projected/0bb2caac-7f37-4362-8257-1fff8220eee2-kube-api-access-qp4tf\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.013809 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.013757 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-tmp-dir\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.014016 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.013999 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-home\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.014235 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.014219 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-model-cache\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.014758 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.014740 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-dshm\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.014966 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.014948 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb2caac-7f37-4362-8257-1fff8220eee2-tls-certs\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.020141 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.020116 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ce3c55\"/\"simulator-single-main-ce3c55-kserve-self-signed-certs\"" Oct 30 17:50:44.021222 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.021197 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ce3c55\"/\"default-dockercfg-nrf4w\"" Oct 30 17:50:44.029105 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.029078 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ce3c55\"/\"openshift-service-ca.crt\"" Oct 30 17:50:44.029243 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.029183 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ce3c55\"/\"kube-root-ca.crt\"" Oct 30 17:50:44.099924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.099428 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:44.116473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.116430 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-dshm\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.116700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.116493 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb2caac-7f37-4362-8257-1fff8220eee2-tls-certs\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.116700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.116538 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp4tf\" (UniqueName: \"kubernetes.io/projected/0bb2caac-7f37-4362-8257-1fff8220eee2-kube-api-access-qp4tf\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.116700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.116595 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-tmp-dir\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.116700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.116620 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-home\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.116700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.116678 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-model-cache\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.117406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.117383 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-tmp-dir\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.117486 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.117439 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-home\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.117723 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.117701 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-model-cache\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.119547 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.119528 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-dshm\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.122555 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.122535 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb2caac-7f37-4362-8257-1fff8220eee2-tls-certs\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.142127 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.142096 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp4tf\" (UniqueName: \"kubernetes.io/projected/0bb2caac-7f37-4362-8257-1fff8220eee2-kube-api-access-qp4tf\") pod \"simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.230986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.230532 4828 patch_prober.go:29] interesting pod/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 17:50:44.230986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.230616 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 17:50:44.324467 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.324423 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:44.476223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.476165 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:44.655179 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.655151 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d"] Oct 30 17:50:44.657400 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:50:44.657363 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bb2caac_7f37_4362_8257_1fff8220eee2.slice/crio-756b0612139d7fdfff46496ec27fd0f3d942f955937701c18e7169e6156b45c4 WatchSource:0}: Error finding container 756b0612139d7fdfff46496ec27fd0f3d942f955937701c18e7169e6156b45c4: Status 404 returned error can't find the container with id 756b0612139d7fdfff46496ec27fd0f3d942f955937701c18e7169e6156b45c4 Oct 30 17:50:44.998027 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.997939 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" event={"ID":"0bb2caac-7f37-4362-8257-1fff8220eee2","Type":"ContainerStarted","Data":"af54e99aeb1c49eb6ea8b9f8afe804da5656098f0e23b8f844ec1ef9b3c583ba"} Oct 30 17:50:44.998027 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:44.997988 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" event={"ID":"0bb2caac-7f37-4362-8257-1fff8220eee2","Type":"ContainerStarted","Data":"756b0612139d7fdfff46496ec27fd0f3d942f955937701c18e7169e6156b45c4"} Oct 30 17:50:45.040705 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:45.040647 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" podStartSLOduration=2.040632582 podStartE2EDuration="2.040632582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:50:45.038333236 +0000 UTC m=+3822.683843410" watchObservedRunningTime="2026-10-30 17:50:45.040632582 +0000 UTC m=+3822.686142759" Oct 30 17:50:45.230365 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:45.230328 4828 patch_prober.go:29] interesting pod/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 17:50:45.230542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:45.230382 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 17:50:45.325461 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:45.325418 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:46.230946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:46.230046 4828 patch_prober.go:29] interesting pod/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 17:50:46.230946 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:46.230096 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 17:50:46.297924 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:46.296615 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:47.230166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:47.230130 4828 patch_prober.go:29] interesting pod/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 30 17:50:47.230350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:47.230186 4828 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 30 17:50:48.158698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:48.158625 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:48.633920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:48.633875 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:48.634395 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:48.634372 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:48.673015 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:48.672985 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:50:49.996515 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:49.996475 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:51.826486 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:51.826448 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:53.677251 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:53.677215 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:54.325053 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:54.325014 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:54.337833 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:54.337806 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:55.076627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:55.076595 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:50:55.337436 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:55.337312 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:56.993978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:56.993937 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:50:58.763872 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:50:58.763834 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:00.441462 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:00.441423 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:01.605423 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:01.605381 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:02.048188 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:02.048146 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:02.251025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:02.250855 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:02.702511 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:02.702471 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:03.973441 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:03.973405 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:05.690307 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:05.690266 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:07.475468 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:07.475420 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:07.911451 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:51:07.911414 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:51:09.117130 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:09.117092 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:10.966662 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:10.966620 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:12.689389 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:12.689346 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:14.039287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:14.039241 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:14.564345 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:14.564308 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:16.501375 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:16.501311 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:18.204055 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:18.204006 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:18.954143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:18.954102 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d"] Oct 30 17:51:19.068853 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:19.068816 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:20.040799 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:20.040751 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:21.790686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:21.790603 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:23.438875 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:23.438824 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:24.301080 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:24.301042 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4"] Oct 30 17:51:24.337497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:24.337453 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p"] Oct 30 17:51:24.497048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:24.497011 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:24.646959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:24.646829 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:24.774617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:24.774574 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k"] Oct 30 17:51:24.774844 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:24.774817 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" podUID="ed6b5197-637f-475c-9c75-4a9277b37746" containerName="maas-api" containerID="cri-o://515fcb92652aa7b0e2342344016bb0135e70c0402c32d17cdb04ee5d261111dd" gracePeriod=30 Oct 30 17:51:24.875123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:24.875079 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:25.075882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.075847 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:25.241950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.241915 4828 generic.go:356] "Generic (PLEG): container finished" podID="ed6b5197-637f-475c-9c75-4a9277b37746" containerID="515fcb92652aa7b0e2342344016bb0135e70c0402c32d17cdb04ee5d261111dd" exitCode=0 Oct 30 17:51:25.242088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.241943 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" event={"ID":"ed6b5197-637f-475c-9c75-4a9277b37746","Type":"ContainerDied","Data":"515fcb92652aa7b0e2342344016bb0135e70c0402c32d17cdb04ee5d261111dd"} Oct 30 17:51:25.262751 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.262722 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:51:25.339617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.339512 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-metrics-certs\") pod \"ed6b5197-637f-475c-9c75-4a9277b37746\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " Oct 30 17:51:25.339617 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.339590 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6b5197-637f-475c-9c75-4a9277b37746-kube-api-access-vlwf2\" (UniqueName: \"kubernetes.io/projected/ed6b5197-637f-475c-9c75-4a9277b37746-kube-api-access-vlwf2\") pod \"ed6b5197-637f-475c-9c75-4a9277b37746\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " Oct 30 17:51:25.339868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.339674 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6b5197-637f-475c-9c75-4a9277b37746-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6b5197-637f-475c-9c75-4a9277b37746-openshift-service-ca\") pod \"ed6b5197-637f-475c-9c75-4a9277b37746\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " Oct 30 17:51:25.339868 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.339716 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-maas-api-tls\") pod \"ed6b5197-637f-475c-9c75-4a9277b37746\" (UID: \"ed6b5197-637f-475c-9c75-4a9277b37746\") " Oct 30 17:51:25.340182 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.340150 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6b5197-637f-475c-9c75-4a9277b37746-openshift-service-ca" pod "ed6b5197-637f-475c-9c75-4a9277b37746" (UID: "ed6b5197-637f-475c-9c75-4a9277b37746"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:51:25.342017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.341876 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-metrics-certs" pod "ed6b5197-637f-475c-9c75-4a9277b37746" (UID: "ed6b5197-637f-475c-9c75-4a9277b37746"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:51:25.342298 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.342278 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-maas-api-tls" pod "ed6b5197-637f-475c-9c75-4a9277b37746" (UID: "ed6b5197-637f-475c-9c75-4a9277b37746"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:51:25.342389 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.342340 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6b5197-637f-475c-9c75-4a9277b37746-kube-api-access-vlwf2" pod "ed6b5197-637f-475c-9c75-4a9277b37746" (UID: "ed6b5197-637f-475c-9c75-4a9277b37746"). InnerVolumeSpecName "kube-api-access-vlwf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:25.441082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.441043 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:25.441082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.441077 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlwf2\" (UniqueName: \"kubernetes.io/projected/ed6b5197-637f-475c-9c75-4a9277b37746-kube-api-access-vlwf2\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:25.441082 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.441087 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed6b5197-637f-475c-9c75-4a9277b37746-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:25.441347 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:25.441101 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed6b5197-637f-475c-9c75-4a9277b37746-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:26.249072 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:26.249039 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" Oct 30 17:51:26.249072 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:26.249067 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k" event={"ID":"ed6b5197-637f-475c-9c75-4a9277b37746","Type":"ContainerDied","Data":"ac884117b1befd1a32d89eb11466344b013f7ab65a6eecab09ee668bba8a0ad5"} Oct 30 17:51:26.249566 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:26.249148 4828 scope.go:122] "RemoveContainer" containerID="515fcb92652aa7b0e2342344016bb0135e70c0402c32d17cdb04ee5d261111dd" Oct 30 17:51:26.298482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:26.298443 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k"] Oct 30 17:51:26.309686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:26.309653 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce3c55-56657b4f57-f4t7k"] Oct 30 17:51:26.490543 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:26.490505 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6b5197-637f-475c-9c75-4a9277b37746" path="/var/lib/kubelet/pods/ed6b5197-637f-475c-9c75-4a9277b37746/volumes" Oct 30 17:51:27.464337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:27.464303 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:27.696818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:27.696780 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:28.133590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:28.133556 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:29.302073 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:29.302013 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" podUID="89d803d3-631d-430b-8a57-3ed320ee6382" containerName="payload-processing" containerID="cri-o://3a6a76f4d85a4b56029a6022937700ec48966c9e5de88e46458c2b7f9b2bfa2a" gracePeriod=25 Oct 30 17:51:29.338149 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:29.338102 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" podUID="1b1e4365-35cf-49ed-b291-4fa18b1411a4" containerName="payload-pre-processing" containerID="cri-o://50e59345235c79f990733eb7705e7d406a59f854ccb54e9fc4d6e8e89f04a779" gracePeriod=25 Oct 30 17:51:30.879675 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:30.879637 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" podUID="89d803d3-631d-430b-8a57-3ed320ee6382" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:51:30.880109 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:30.879694 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" podUID="1b1e4365-35cf-49ed-b291-4fa18b1411a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:51:33.975565 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:33.975503 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" podUID="0bb2caac-7f37-4362-8257-1fff8220eee2" containerName="main" containerID="cri-o://af54e99aeb1c49eb6ea8b9f8afe804da5656098f0e23b8f844ec1ef9b3c583ba" gracePeriod=45 Oct 30 17:51:34.003838 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.003804 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:34.066586 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.066544 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" podUID="0bb2caac-7f37-4362-8257-1fff8220eee2" containerName="main" probeResult="failure" output="Get \"https://10.132.1.0:8000/ready\": dial tcp 10.132.1.0:8000: connect: connection refused" Oct 30 17:51:34.223497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.223450 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:34.233686 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.233628 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:51:34.292646 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.292607 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb"] Oct 30 17:51:34.292860 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.292835 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerName="istio-proxy" containerID="cri-o://7251241bb2f4dd45669a2077a55ee1d41b9fb1ed531499aba794eeb8e68f374e" gracePeriod=30 Oct 30 17:51:34.303061 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.303031 4828 generic.go:356] "Generic (PLEG): container finished" podID="0bb2caac-7f37-4362-8257-1fff8220eee2" containerID="af54e99aeb1c49eb6ea8b9f8afe804da5656098f0e23b8f844ec1ef9b3c583ba" exitCode=0 Oct 30 17:51:34.303848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.303078 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" Oct 30 17:51:34.303848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.303094 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" event={"ID":"0bb2caac-7f37-4362-8257-1fff8220eee2","Type":"ContainerDied","Data":"af54e99aeb1c49eb6ea8b9f8afe804da5656098f0e23b8f844ec1ef9b3c583ba"} Oct 30 17:51:34.303848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.303430 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d" event={"ID":"0bb2caac-7f37-4362-8257-1fff8220eee2","Type":"ContainerDied","Data":"756b0612139d7fdfff46496ec27fd0f3d942f955937701c18e7169e6156b45c4"} Oct 30 17:51:34.303848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.303461 4828 scope.go:122] "RemoveContainer" containerID="af54e99aeb1c49eb6ea8b9f8afe804da5656098f0e23b8f844ec1ef9b3c583ba" Oct 30 17:51:34.317002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.316978 4828 scope.go:122] "RemoveContainer" containerID="af54e99aeb1c49eb6ea8b9f8afe804da5656098f0e23b8f844ec1ef9b3c583ba" Oct 30 17:51:34.317333 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:51:34.317313 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af54e99aeb1c49eb6ea8b9f8afe804da5656098f0e23b8f844ec1ef9b3c583ba\": container with ID starting with af54e99aeb1c49eb6ea8b9f8afe804da5656098f0e23b8f844ec1ef9b3c583ba not found: ID does not exist" containerID="af54e99aeb1c49eb6ea8b9f8afe804da5656098f0e23b8f844ec1ef9b3c583ba" Oct 30 17:51:34.317388 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.317344 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af54e99aeb1c49eb6ea8b9f8afe804da5656098f0e23b8f844ec1ef9b3c583ba"} err="failed to get container status \"af54e99aeb1c49eb6ea8b9f8afe804da5656098f0e23b8f844ec1ef9b3c583ba\": rpc error: code = NotFound desc = could not find container \"af54e99aeb1c49eb6ea8b9f8afe804da5656098f0e23b8f844ec1ef9b3c583ba\": container with ID starting with af54e99aeb1c49eb6ea8b9f8afe804da5656098f0e23b8f844ec1ef9b3c583ba not found: ID does not exist" Oct 30 17:51:34.331679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.331649 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb2caac-7f37-4362-8257-1fff8220eee2-kube-api-access-qp4tf\" (UniqueName: \"kubernetes.io/projected/0bb2caac-7f37-4362-8257-1fff8220eee2-kube-api-access-qp4tf\") pod \"0bb2caac-7f37-4362-8257-1fff8220eee2\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " Oct 30 17:51:34.331842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.331719 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-home\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-home\") pod \"0bb2caac-7f37-4362-8257-1fff8220eee2\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " Oct 30 17:51:34.331842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.331765 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-dshm\") pod \"0bb2caac-7f37-4362-8257-1fff8220eee2\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " Oct 30 17:51:34.331842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.331814 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-model-cache\") pod \"0bb2caac-7f37-4362-8257-1fff8220eee2\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " Oct 30 17:51:34.332123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.331889 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bb2caac-7f37-4362-8257-1fff8220eee2-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb2caac-7f37-4362-8257-1fff8220eee2-tls-certs\") pod \"0bb2caac-7f37-4362-8257-1fff8220eee2\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " Oct 30 17:51:34.332123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.331971 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-tmp-dir\") pod \"0bb2caac-7f37-4362-8257-1fff8220eee2\" (UID: \"0bb2caac-7f37-4362-8257-1fff8220eee2\") " Oct 30 17:51:34.332123 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.332014 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-home" pod "0bb2caac-7f37-4362-8257-1fff8220eee2" (UID: "0bb2caac-7f37-4362-8257-1fff8220eee2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:34.332323 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.332259 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-model-cache" pod "0bb2caac-7f37-4362-8257-1fff8220eee2" (UID: "0bb2caac-7f37-4362-8257-1fff8220eee2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:34.332323 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.332270 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-tmp-dir" pod "0bb2caac-7f37-4362-8257-1fff8220eee2" (UID: "0bb2caac-7f37-4362-8257-1fff8220eee2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:34.332446 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.332316 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:34.334118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.334095 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb2caac-7f37-4362-8257-1fff8220eee2-tls-certs" pod "0bb2caac-7f37-4362-8257-1fff8220eee2" (UID: "0bb2caac-7f37-4362-8257-1fff8220eee2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:51:34.334392 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.334370 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-dshm" pod "0bb2caac-7f37-4362-8257-1fff8220eee2" (UID: "0bb2caac-7f37-4362-8257-1fff8220eee2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:34.334496 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.334404 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb2caac-7f37-4362-8257-1fff8220eee2-kube-api-access-qp4tf" pod "0bb2caac-7f37-4362-8257-1fff8220eee2" (UID: "0bb2caac-7f37-4362-8257-1fff8220eee2"). InnerVolumeSpecName "kube-api-access-qp4tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:34.379112 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.379082 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:34.433625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.433592 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:34.433625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.433622 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:34.433625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.433632 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bb2caac-7f37-4362-8257-1fff8220eee2-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:34.433625 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.433640 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bb2caac-7f37-4362-8257-1fff8220eee2-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:34.433948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.433650 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp4tf\" (UniqueName: \"kubernetes.io/projected/0bb2caac-7f37-4362-8257-1fff8220eee2-kube-api-access-qp4tf\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:34.644656 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.644617 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d"] Oct 30 17:51:34.648819 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:34.648793 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ce3c55/simulator-single-main-ce3c55-kserve-55ddf7df78-m6l2d"] Oct 30 17:51:36.490957 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:36.490924 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb2caac-7f37-4362-8257-1fff8220eee2" path="/var/lib/kubelet/pods/0bb2caac-7f37-4362-8257-1fff8220eee2/volumes" Oct 30 17:51:37.910321 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:51:37.910278 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:51:38.662636 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.662601 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj"] Oct 30 17:51:38.663157 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.663136 4828 cpu_manager.go:414] "containerMap: removing container" podUID="0bb2caac-7f37-4362-8257-1fff8220eee2" containerName="main" Oct 30 17:51:38.663275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.663159 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb2caac-7f37-4362-8257-1fff8220eee2" containerName="main" Oct 30 17:51:38.663275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.663232 4828 cpu_manager.go:414] "containerMap: removing container" podUID="ed6b5197-637f-475c-9c75-4a9277b37746" containerName="maas-api" Oct 30 17:51:38.663275 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.663241 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6b5197-637f-475c-9c75-4a9277b37746" containerName="maas-api" Oct 30 17:51:38.663437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.663366 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb2caac-7f37-4362-8257-1fff8220eee2" containerName="main" Oct 30 17:51:38.663437 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.663391 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6b5197-637f-475c-9c75-4a9277b37746" containerName="maas-api" Oct 30 17:51:38.668086 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.668059 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.696372 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.696327 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b0591e0c-kserve-self-signed-certs\"" Oct 30 17:51:38.699058 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.699030 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj"] Oct 30 17:51:38.755526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.755487 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:38.778606 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.778569 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tmp-dir\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.778784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.778623 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-home\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.778784 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.778755 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-model-cache\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.778878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.778804 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-dshm\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.778878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.778851 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tls-certs\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.779032 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.778943 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjj8\" (UniqueName: \"kubernetes.io/projected/e2a822dd-07c1-4ad7-8b34-f1f573199e53-kube-api-access-hwjj8\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.879948 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.879876 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-model-cache\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.880132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.879978 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-dshm\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.880132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.880027 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tls-certs\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.880132 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.880075 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjj8\" (UniqueName: \"kubernetes.io/projected/e2a822dd-07c1-4ad7-8b34-f1f573199e53-kube-api-access-hwjj8\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.880264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.880148 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tmp-dir\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.880264 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.880210 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-home\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.880349 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.880289 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-model-cache\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.880473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.880447 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tmp-dir\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.880612 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.880545 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-home\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.882265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.882246 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-dshm\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.882547 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.882529 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tls-certs\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.924700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.924633 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjj8\" (UniqueName: \"kubernetes.io/projected/e2a822dd-07c1-4ad7-8b34-f1f573199e53-kube-api-access-hwjj8\") pod \"e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:38.977328 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.977289 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:38.983209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:38.983186 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:39.286209 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.286181 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj"] Oct 30 17:51:39.288402 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:51:39.288358 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2a822dd_07c1_4ad7_8b34_f1f573199e53.slice/crio-840b13eefa74d334fad72b4178124c2571c6d6ec50369c891cea3b3af9a462c2 WatchSource:0}: Error finding container 840b13eefa74d334fad72b4178124c2571c6d6ec50369c891cea3b3af9a462c2: Status 404 returned error can't find the container with id 840b13eefa74d334fad72b4178124c2571c6d6ec50369c891cea3b3af9a462c2 Oct 30 17:51:39.337510 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.337473 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" event={"ID":"e2a822dd-07c1-4ad7-8b34-f1f573199e53","Type":"ContainerStarted","Data":"840b13eefa74d334fad72b4178124c2571c6d6ec50369c891cea3b3af9a462c2"} Oct 30 17:51:39.540992 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.540933 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:51:39.586859 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.586824 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-podinfo\") pod \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " Oct 30 17:51:39.587081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.586930 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-data\") pod \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " Oct 30 17:51:39.587081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.586976 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-certs\") pod \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " Oct 30 17:51:39.587081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.586999 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-credential-socket\") pod \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " Oct 30 17:51:39.587081 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.587061 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-token\" (UniqueName: \"kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-token\") pod \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " Oct 30 17:51:39.587338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.587149 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-kube-api-access-bvf7d\" (UniqueName: \"kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-kube-api-access-bvf7d\") pod \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " Oct 30 17:51:39.587338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.587200 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-socket\") pod \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " Oct 30 17:51:39.587338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.587248 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istiod-ca-cert\") pod \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " Oct 30 17:51:39.587338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.587276 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-envoy\") pod \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\" (UID: \"c5c0b5e6-73ab-4398-a200-c7ef5e974373\") " Oct 30 17:51:39.587338 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.587321 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-certs" pod "c5c0b5e6-73ab-4398-a200-c7ef5e974373" (UID: "c5c0b5e6-73ab-4398-a200-c7ef5e974373"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:39.587600 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.587346 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-data" pod "c5c0b5e6-73ab-4398-a200-c7ef5e974373" (UID: "c5c0b5e6-73ab-4398-a200-c7ef5e974373"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:39.587600 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.587551 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-socket" pod "c5c0b5e6-73ab-4398-a200-c7ef5e974373" (UID: "c5c0b5e6-73ab-4398-a200-c7ef5e974373"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:39.587719 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.587659 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istiod-ca-cert" pod "c5c0b5e6-73ab-4398-a200-c7ef5e974373" (UID: "c5c0b5e6-73ab-4398-a200-c7ef5e974373"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:51:39.587778 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.587734 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-credential-socket" pod "c5c0b5e6-73ab-4398-a200-c7ef5e974373" (UID: "c5c0b5e6-73ab-4398-a200-c7ef5e974373"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:39.588108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.587994 4828 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:39.588108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.588035 4828 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istiod-ca-cert\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:39.588108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.588050 4828 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-data\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:39.588108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.588064 4828 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-workload-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:39.589490 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.589454 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-podinfo" pod "c5c0b5e6-73ab-4398-a200-c7ef5e974373" (UID: "c5c0b5e6-73ab-4398-a200-c7ef5e974373"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:51:39.590488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.590383 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-kube-api-access-bvf7d" pod "c5c0b5e6-73ab-4398-a200-c7ef5e974373" (UID: "c5c0b5e6-73ab-4398-a200-c7ef5e974373"). InnerVolumeSpecName "kube-api-access-bvf7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:39.590488 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.590462 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-envoy" pod "c5c0b5e6-73ab-4398-a200-c7ef5e974373" (UID: "c5c0b5e6-73ab-4398-a200-c7ef5e974373"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:51:39.591385 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.591362 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-token" pod "c5c0b5e6-73ab-4398-a200-c7ef5e974373" (UID: "c5c0b5e6-73ab-4398-a200-c7ef5e974373"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:39.689060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.689024 4828 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-credential-socket\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:39.689060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.689054 4828 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-token\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:39.689060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.689064 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvf7d\" (UniqueName: \"kubernetes.io/projected/c5c0b5e6-73ab-4398-a200-c7ef5e974373-kube-api-access-bvf7d\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:39.689313 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.689074 4828 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-envoy\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:39.689313 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:39.689084 4828 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5c0b5e6-73ab-4398-a200-c7ef5e974373-istio-podinfo\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:40.345177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.345139 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" event={"ID":"e2a822dd-07c1-4ad7-8b34-f1f573199e53","Type":"ContainerStarted","Data":"cd396bcf56178af59146cca67ca855263f550aac5be7ffdac1b2e3d2f3c93680"} Oct 30 17:51:40.346956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.346896 4828 generic.go:356] "Generic (PLEG): container finished" podID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerID="7251241bb2f4dd45669a2077a55ee1d41b9fb1ed531499aba794eeb8e68f374e" exitCode=0 Oct 30 17:51:40.346956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.346936 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" event={"ID":"c5c0b5e6-73ab-4398-a200-c7ef5e974373","Type":"ContainerDied","Data":"7251241bb2f4dd45669a2077a55ee1d41b9fb1ed531499aba794eeb8e68f374e"} Oct 30 17:51:40.346956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.346952 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" Oct 30 17:51:40.347211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.346995 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb" event={"ID":"c5c0b5e6-73ab-4398-a200-c7ef5e974373","Type":"ContainerDied","Data":"037e7b9e7cf13419bbddb3186148575db10c6e458da881c8d01bc72a8b904ec6"} Oct 30 17:51:40.347211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.347029 4828 scope.go:122] "RemoveContainer" containerID="7251241bb2f4dd45669a2077a55ee1d41b9fb1ed531499aba794eeb8e68f374e" Oct 30 17:51:40.356710 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.356689 4828 scope.go:122] "RemoveContainer" containerID="06bdfc415363bdfd93c272d903eeb10f4e3dae4aab670ceec963a78408bcf7bf" Oct 30 17:51:40.365292 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.365272 4828 scope.go:122] "RemoveContainer" containerID="7251241bb2f4dd45669a2077a55ee1d41b9fb1ed531499aba794eeb8e68f374e" Oct 30 17:51:40.365570 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:51:40.365546 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7251241bb2f4dd45669a2077a55ee1d41b9fb1ed531499aba794eeb8e68f374e\": container with ID starting with 7251241bb2f4dd45669a2077a55ee1d41b9fb1ed531499aba794eeb8e68f374e not found: ID does not exist" containerID="7251241bb2f4dd45669a2077a55ee1d41b9fb1ed531499aba794eeb8e68f374e" Oct 30 17:51:40.365668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.365583 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7251241bb2f4dd45669a2077a55ee1d41b9fb1ed531499aba794eeb8e68f374e"} err="failed to get container status \"7251241bb2f4dd45669a2077a55ee1d41b9fb1ed531499aba794eeb8e68f374e\": rpc error: code = NotFound desc = could not find container \"7251241bb2f4dd45669a2077a55ee1d41b9fb1ed531499aba794eeb8e68f374e\": container with ID starting with 7251241bb2f4dd45669a2077a55ee1d41b9fb1ed531499aba794eeb8e68f374e not found: ID does not exist" Oct 30 17:51:40.365668 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.365606 4828 scope.go:122] "RemoveContainer" containerID="06bdfc415363bdfd93c272d903eeb10f4e3dae4aab670ceec963a78408bcf7bf" Oct 30 17:51:40.365859 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:51:40.365841 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bdfc415363bdfd93c272d903eeb10f4e3dae4aab670ceec963a78408bcf7bf\": container with ID starting with 06bdfc415363bdfd93c272d903eeb10f4e3dae4aab670ceec963a78408bcf7bf not found: ID does not exist" containerID="06bdfc415363bdfd93c272d903eeb10f4e3dae4aab670ceec963a78408bcf7bf" Oct 30 17:51:40.366010 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.365866 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bdfc415363bdfd93c272d903eeb10f4e3dae4aab670ceec963a78408bcf7bf"} err="failed to get container status \"06bdfc415363bdfd93c272d903eeb10f4e3dae4aab670ceec963a78408bcf7bf\": rpc error: code = NotFound desc = could not find container \"06bdfc415363bdfd93c272d903eeb10f4e3dae4aab670ceec963a78408bcf7bf\": container with ID starting with 06bdfc415363bdfd93c272d903eeb10f4e3dae4aab670ceec963a78408bcf7bf not found: ID does not exist" Oct 30 17:51:40.387610 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.387558 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" podStartSLOduration=2.387542822 podStartE2EDuration="2.387542822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:40.38614619 +0000 UTC m=+3878.031656392" watchObservedRunningTime="2026-10-30 17:51:40.387542822 +0000 UTC m=+3878.033053003" Oct 30 17:51:40.423178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.423143 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb"] Oct 30 17:51:40.434825 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.434785 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ce3c55-openshift-default-76f4d9fd49xzmmb"] Oct 30 17:51:40.491508 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.491475 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" path="/var/lib/kubelet/pods/c5c0b5e6-73ab-4398-a200-c7ef5e974373/volumes" Oct 30 17:51:40.879979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.879938 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" podUID="1b1e4365-35cf-49ed-b291-4fa18b1411a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:51:40.880221 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.879936 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" podUID="89d803d3-631d-430b-8a57-3ed320ee6382" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:51:40.984002 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:40.983965 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:48.984331 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:48.984296 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:48.997224 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:48.997189 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:49.476397 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.476361 4828 generic.go:356] "Generic (PLEG): container finished" podID="1b1e4365-35cf-49ed-b291-4fa18b1411a4" containerID="50e59345235c79f990733eb7705e7d406a59f854ccb54e9fc4d6e8e89f04a779" exitCode=0 Oct 30 17:51:49.476589 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.476406 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" event={"ID":"1b1e4365-35cf-49ed-b291-4fa18b1411a4","Type":"ContainerDied","Data":"50e59345235c79f990733eb7705e7d406a59f854ccb54e9fc4d6e8e89f04a779"} Oct 30 17:51:49.478476 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.478450 4828 generic.go:356] "Generic (PLEG): container finished" podID="89d803d3-631d-430b-8a57-3ed320ee6382" containerID="3a6a76f4d85a4b56029a6022937700ec48966c9e5de88e46458c2b7f9b2bfa2a" exitCode=0 Oct 30 17:51:49.478627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.478525 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" event={"ID":"89d803d3-631d-430b-8a57-3ed320ee6382","Type":"ContainerDied","Data":"3a6a76f4d85a4b56029a6022937700ec48966c9e5de88e46458c2b7f9b2bfa2a"} Oct 30 17:51:49.493540 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.493510 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:51:49.612466 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.612429 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" Oct 30 17:51:49.634952 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.634923 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" Oct 30 17:51:49.787915 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.787872 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d803d3-631d-430b-8a57-3ed320ee6382-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d803d3-631d-430b-8a57-3ed320ee6382-plugins-config-volume\") pod \"89d803d3-631d-430b-8a57-3ed320ee6382\" (UID: \"89d803d3-631d-430b-8a57-3ed320ee6382\") " Oct 30 17:51:49.788104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.787973 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d803d3-631d-430b-8a57-3ed320ee6382-kube-api-access-sfjrr\" (UniqueName: \"kubernetes.io/projected/89d803d3-631d-430b-8a57-3ed320ee6382-kube-api-access-sfjrr\") pod \"89d803d3-631d-430b-8a57-3ed320ee6382\" (UID: \"89d803d3-631d-430b-8a57-3ed320ee6382\") " Oct 30 17:51:49.788104 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.788089 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1e4365-35cf-49ed-b291-4fa18b1411a4-kube-api-access-52kcp\" (UniqueName: \"kubernetes.io/projected/1b1e4365-35cf-49ed-b291-4fa18b1411a4-kube-api-access-52kcp\") pod \"1b1e4365-35cf-49ed-b291-4fa18b1411a4\" (UID: \"1b1e4365-35cf-49ed-b291-4fa18b1411a4\") " Oct 30 17:51:49.788213 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.788167 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1e4365-35cf-49ed-b291-4fa18b1411a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1e4365-35cf-49ed-b291-4fa18b1411a4-plugins-config-volume\") pod \"1b1e4365-35cf-49ed-b291-4fa18b1411a4\" (UID: \"1b1e4365-35cf-49ed-b291-4fa18b1411a4\") " Oct 30 17:51:49.788287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.788263 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d803d3-631d-430b-8a57-3ed320ee6382-plugins-config-volume" pod "89d803d3-631d-430b-8a57-3ed320ee6382" (UID: "89d803d3-631d-430b-8a57-3ed320ee6382"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:51:49.788569 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.788543 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1e4365-35cf-49ed-b291-4fa18b1411a4-plugins-config-volume" pod "1b1e4365-35cf-49ed-b291-4fa18b1411a4" (UID: "1b1e4365-35cf-49ed-b291-4fa18b1411a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:51:49.788698 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.788589 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d803d3-631d-430b-8a57-3ed320ee6382-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:49.790254 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.790225 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d803d3-631d-430b-8a57-3ed320ee6382-kube-api-access-sfjrr" pod "89d803d3-631d-430b-8a57-3ed320ee6382" (UID: "89d803d3-631d-430b-8a57-3ed320ee6382"). InnerVolumeSpecName "kube-api-access-sfjrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:49.790350 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.790296 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1e4365-35cf-49ed-b291-4fa18b1411a4-kube-api-access-52kcp" pod "1b1e4365-35cf-49ed-b291-4fa18b1411a4" (UID: "1b1e4365-35cf-49ed-b291-4fa18b1411a4"). InnerVolumeSpecName "kube-api-access-52kcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:51:49.889665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.889578 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52kcp\" (UniqueName: \"kubernetes.io/projected/1b1e4365-35cf-49ed-b291-4fa18b1411a4-kube-api-access-52kcp\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:49.889665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.889613 4828 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b1e4365-35cf-49ed-b291-4fa18b1411a4-plugins-config-volume\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:49.889665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:49.889622 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfjrr\" (UniqueName: \"kubernetes.io/projected/89d803d3-631d-430b-8a57-3ed320ee6382-kube-api-access-sfjrr\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:51:50.486156 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:50.486126 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" Oct 30 17:51:50.487878 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:50.487852 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" Oct 30 17:51:50.488526 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:50.488497 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p" event={"ID":"1b1e4365-35cf-49ed-b291-4fa18b1411a4","Type":"ContainerDied","Data":"acfbba415d91e0ff3735e8993a4b917648d12f6709e4fc697b517094ccef31e2"} Oct 30 17:51:50.488665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:50.488550 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4" event={"ID":"89d803d3-631d-430b-8a57-3ed320ee6382","Type":"ContainerDied","Data":"cb482d32ab2bd0ed64c1bbddfd11bf964b67cdd88482d2001a44e8ea1cf4764b"} Oct 30 17:51:50.488665 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:50.488583 4828 scope.go:122] "RemoveContainer" containerID="50e59345235c79f990733eb7705e7d406a59f854ccb54e9fc4d6e8e89f04a779" Oct 30 17:51:50.502935 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:50.502885 4828 scope.go:122] "RemoveContainer" containerID="3a6a76f4d85a4b56029a6022937700ec48966c9e5de88e46458c2b7f9b2bfa2a" Oct 30 17:51:50.550088 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:50.550054 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p"] Oct 30 17:51:50.554771 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:50.554742 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ce3c55-5cc98gjw2p"] Oct 30 17:51:50.594139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:50.594059 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4"] Oct 30 17:51:50.598843 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:50.598818 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ce3c55-854b8fc7fbvzn4"] Oct 30 17:51:52.489627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:52.489585 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1e4365-35cf-49ed-b291-4fa18b1411a4" path="/var/lib/kubelet/pods/1b1e4365-35cf-49ed-b291-4fa18b1411a4/volumes" Oct 30 17:51:52.490142 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:52.490072 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d803d3-631d-430b-8a57-3ed320ee6382" path="/var/lib/kubelet/pods/89d803d3-631d-430b-8a57-3ed320ee6382/volumes" Oct 30 17:51:55.450980 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:55.450356 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:56.732442 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.732394 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:56.777986 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.777952 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l"] Oct 30 17:51:56.778669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.778643 4828 cpu_manager.go:414] "containerMap: removing container" podUID="89d803d3-631d-430b-8a57-3ed320ee6382" containerName="payload-processing" Oct 30 17:51:56.778669 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.778666 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d803d3-631d-430b-8a57-3ed320ee6382" containerName="payload-processing" Oct 30 17:51:56.778812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.778686 4828 cpu_manager.go:414] "containerMap: removing container" podUID="1b1e4365-35cf-49ed-b291-4fa18b1411a4" containerName="payload-pre-processing" Oct 30 17:51:56.778812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.778692 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1e4365-35cf-49ed-b291-4fa18b1411a4" containerName="payload-pre-processing" Oct 30 17:51:56.778812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.778703 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerName="istio-proxy" Oct 30 17:51:56.778812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.778708 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerName="istio-proxy" Oct 30 17:51:56.778812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.778716 4828 cpu_manager.go:414] "containerMap: removing container" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerName="istio-proxy" Oct 30 17:51:56.778812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.778721 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerName="istio-proxy" Oct 30 17:51:56.778812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.778798 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d803d3-631d-430b-8a57-3ed320ee6382" containerName="payload-processing" Oct 30 17:51:56.778812 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.778811 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1e4365-35cf-49ed-b291-4fa18b1411a4" containerName="payload-pre-processing" Oct 30 17:51:56.779107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.778819 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerName="istio-proxy" Oct 30 17:51:56.779107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.778825 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c0b5e6-73ab-4398-a200-c7ef5e974373" containerName="istio-proxy" Oct 30 17:51:56.789963 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.789930 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.804837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.804806 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l"] Oct 30 17:51:56.811746 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.811703 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-b0591e0c-kserve-self-signed-certs\"" Oct 30 17:51:56.858636 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.858598 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-home\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.858828 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.858652 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tls-certs\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.858828 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.858677 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tmp-dir\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.858934 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.858842 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x62fr\" (UniqueName: \"kubernetes.io/projected/a8940fb3-07af-4f6b-90a4-afdd309b6b69-kube-api-access-x62fr\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.858981 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.858935 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-dshm\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.859025 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.858996 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-model-cache\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.900767 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.900731 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:51:56.960096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.960060 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-model-cache\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.960096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.960106 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-home\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.960384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.960164 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tls-certs\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.960384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.960192 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tmp-dir\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.960384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.960276 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x62fr\" (UniqueName: \"kubernetes.io/projected/a8940fb3-07af-4f6b-90a4-afdd309b6b69-kube-api-access-x62fr\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.960384 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.960337 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-dshm\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.961400 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.961367 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-model-cache\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.961548 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.961516 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tmp-dir\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.961649 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.961598 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-home\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.963515 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.963485 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-dshm\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.970794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.970762 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tls-certs\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:56.988794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:56.988707 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x62fr\" (UniqueName: \"kubernetes.io/projected/a8940fb3-07af-4f6b-90a4-afdd309b6b69-kube-api-access-x62fr\") pod \"e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:57.106178 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:57.106139 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:51:57.417121 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:57.417077 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l"] Oct 30 17:51:57.418835 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:51:57.418806 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8940fb3_07af_4f6b_90a4_afdd309b6b69.slice/crio-b829f0c6f8014dfbcf4a10a3407e507b5ffd21b4e0c1deac6369dc5a93151b24 WatchSource:0}: Error finding container b829f0c6f8014dfbcf4a10a3407e507b5ffd21b4e0c1deac6369dc5a93151b24: Status 404 returned error can't find the container with id b829f0c6f8014dfbcf4a10a3407e507b5ffd21b4e0c1deac6369dc5a93151b24 Oct 30 17:51:57.540827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:57.540772 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" event={"ID":"a8940fb3-07af-4f6b-90a4-afdd309b6b69","Type":"ContainerStarted","Data":"e12868ab2ecabe7977f26aca841bda3e9d4108e3230201392bd8a9dce1ff4a62"} Oct 30 17:51:57.540827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:57.540832 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" event={"ID":"a8940fb3-07af-4f6b-90a4-afdd309b6b69","Type":"ContainerStarted","Data":"b829f0c6f8014dfbcf4a10a3407e507b5ffd21b4e0c1deac6369dc5a93151b24"} Oct 30 17:51:57.585822 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:57.585395 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" podStartSLOduration=1.585375484 podStartE2EDuration="1.585375484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:51:57.581012435 +0000 UTC m=+3895.226522615" watchObservedRunningTime="2026-10-30 17:51:57.585375484 +0000 UTC m=+3895.230885665" Oct 30 17:51:58.106448 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:51:58.106402 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:52:02.335628 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:02.335570 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:02.727975 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:02.725617 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:07.106671 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:07.106631 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:52:07.119139 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:07.119111 4828 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:52:07.664329 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:07.664298 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:52:07.911094 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:52:07.911052 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:52:08.074212 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:08.074183 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:52:08.084716 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:08.084688 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:52:25.996622 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:25.996524 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj"] Oct 30 17:52:26.085700 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:26.085665 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:31.364166 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:31.364121 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:52:31.375057 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:31.368663 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l"] Oct 30 17:52:35.316923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:35.316007 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-22qj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:52:35.316923 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:35.316105 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-22qj7" podUID="54ac60a7-ad93-4811-8242-5f28289f5d1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:52:38.080227 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:52:38.080184 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:52:41.018210 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.018166 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" podUID="e2a822dd-07c1-4ad7-8b34-f1f573199e53" containerName="main" containerID="cri-o://cd396bcf56178af59146cca67ca855263f550aac5be7ffdac1b2e3d2f3c93680" gracePeriod=45 Oct 30 17:52:41.279164 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.279097 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:52:41.360614 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.360575 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-model-cache\") pod \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " Oct 30 17:52:41.360798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.360631 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-dshm\") pod \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " Oct 30 17:52:41.360798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.360653 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-home\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-home\") pod \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " Oct 30 17:52:41.360798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.360682 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a822dd-07c1-4ad7-8b34-f1f573199e53-kube-api-access-hwjj8\" (UniqueName: \"kubernetes.io/projected/e2a822dd-07c1-4ad7-8b34-f1f573199e53-kube-api-access-hwjj8\") pod \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " Oct 30 17:52:41.360798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.360734 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tmp-dir\") pod \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " Oct 30 17:52:41.360798 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.360753 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tls-certs\") pod \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\" (UID: \"e2a822dd-07c1-4ad7-8b34-f1f573199e53\") " Oct 30 17:52:41.361107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.360958 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-model-cache" pod "e2a822dd-07c1-4ad7-8b34-f1f573199e53" (UID: "e2a822dd-07c1-4ad7-8b34-f1f573199e53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:41.361107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.361049 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-home" pod "e2a822dd-07c1-4ad7-8b34-f1f573199e53" (UID: "e2a822dd-07c1-4ad7-8b34-f1f573199e53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:41.361107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.361076 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tmp-dir" pod "e2a822dd-07c1-4ad7-8b34-f1f573199e53" (UID: "e2a822dd-07c1-4ad7-8b34-f1f573199e53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:41.363170 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.363139 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-dshm" pod "e2a822dd-07c1-4ad7-8b34-f1f573199e53" (UID: "e2a822dd-07c1-4ad7-8b34-f1f573199e53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:41.363280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.363239 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a822dd-07c1-4ad7-8b34-f1f573199e53-kube-api-access-hwjj8" pod "e2a822dd-07c1-4ad7-8b34-f1f573199e53" (UID: "e2a822dd-07c1-4ad7-8b34-f1f573199e53"). InnerVolumeSpecName "kube-api-access-hwjj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:52:41.363280 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.363236 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tls-certs" pod "e2a822dd-07c1-4ad7-8b34-f1f573199e53" (UID: "e2a822dd-07c1-4ad7-8b34-f1f573199e53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:52:41.461809 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.461768 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:52:41.461809 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.461808 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:52:41.462065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.461820 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:52:41.462065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.461834 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwjj8\" (UniqueName: \"kubernetes.io/projected/e2a822dd-07c1-4ad7-8b34-f1f573199e53-kube-api-access-hwjj8\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:52:41.462065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.461847 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:52:41.462065 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:41.461860 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2a822dd-07c1-4ad7-8b34-f1f573199e53-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:52:42.029593 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:42.029556 4828 generic.go:356] "Generic (PLEG): container finished" podID="e2a822dd-07c1-4ad7-8b34-f1f573199e53" containerID="cd396bcf56178af59146cca67ca855263f550aac5be7ffdac1b2e3d2f3c93680" exitCode=0 Oct 30 17:52:42.030107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:42.029631 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" Oct 30 17:52:42.030107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:42.029638 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" event={"ID":"e2a822dd-07c1-4ad7-8b34-f1f573199e53","Type":"ContainerDied","Data":"cd396bcf56178af59146cca67ca855263f550aac5be7ffdac1b2e3d2f3c93680"} Oct 30 17:52:42.030107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:42.029692 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj" event={"ID":"e2a822dd-07c1-4ad7-8b34-f1f573199e53","Type":"ContainerDied","Data":"840b13eefa74d334fad72b4178124c2571c6d6ec50369c891cea3b3af9a462c2"} Oct 30 17:52:42.030107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:42.029715 4828 scope.go:122] "RemoveContainer" containerID="cd396bcf56178af59146cca67ca855263f550aac5be7ffdac1b2e3d2f3c93680" Oct 30 17:52:42.039808 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:42.039786 4828 scope.go:122] "RemoveContainer" containerID="cd396bcf56178af59146cca67ca855263f550aac5be7ffdac1b2e3d2f3c93680" Oct 30 17:52:42.040132 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:52:42.040109 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd396bcf56178af59146cca67ca855263f550aac5be7ffdac1b2e3d2f3c93680\": container with ID starting with cd396bcf56178af59146cca67ca855263f550aac5be7ffdac1b2e3d2f3c93680 not found: ID does not exist" containerID="cd396bcf56178af59146cca67ca855263f550aac5be7ffdac1b2e3d2f3c93680" Oct 30 17:52:42.040201 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:42.040141 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd396bcf56178af59146cca67ca855263f550aac5be7ffdac1b2e3d2f3c93680"} err="failed to get container status \"cd396bcf56178af59146cca67ca855263f550aac5be7ffdac1b2e3d2f3c93680\": rpc error: code = NotFound desc = could not find container \"cd396bcf56178af59146cca67ca855263f550aac5be7ffdac1b2e3d2f3c93680\": container with ID starting with cd396bcf56178af59146cca67ca855263f550aac5be7ffdac1b2e3d2f3c93680 not found: ID does not exist" Oct 30 17:52:42.082020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:42.081987 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj"] Oct 30 17:52:42.085287 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:42.085264 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b0591e0c-kserve-7b7b5dddd9-m4vkj"] Oct 30 17:52:42.489773 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:42.489685 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a822dd-07c1-4ad7-8b34-f1f573199e53" path="/var/lib/kubelet/pods/e2a822dd-07c1-4ad7-8b34-f1f573199e53/volumes" Oct 30 17:52:46.743223 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:46.743176 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" podUID="a8940fb3-07af-4f6b-90a4-afdd309b6b69" containerName="main" containerID="cri-o://e12868ab2ecabe7977f26aca841bda3e9d4108e3230201392bd8a9dce1ff4a62" gracePeriod=45 Oct 30 17:52:47.018309 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.018283 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:52:47.061664 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.061631 4828 generic.go:356] "Generic (PLEG): container finished" podID="a8940fb3-07af-4f6b-90a4-afdd309b6b69" containerID="e12868ab2ecabe7977f26aca841bda3e9d4108e3230201392bd8a9dce1ff4a62" exitCode=0 Oct 30 17:52:47.061879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.061681 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" Oct 30 17:52:47.061879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.061719 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" event={"ID":"a8940fb3-07af-4f6b-90a4-afdd309b6b69","Type":"ContainerDied","Data":"e12868ab2ecabe7977f26aca841bda3e9d4108e3230201392bd8a9dce1ff4a62"} Oct 30 17:52:47.061879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.061775 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l" event={"ID":"a8940fb3-07af-4f6b-90a4-afdd309b6b69","Type":"ContainerDied","Data":"b829f0c6f8014dfbcf4a10a3407e507b5ffd21b4e0c1deac6369dc5a93151b24"} Oct 30 17:52:47.061879 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.061797 4828 scope.go:122] "RemoveContainer" containerID="e12868ab2ecabe7977f26aca841bda3e9d4108e3230201392bd8a9dce1ff4a62" Oct 30 17:52:47.081609 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.081573 4828 scope.go:122] "RemoveContainer" containerID="e12868ab2ecabe7977f26aca841bda3e9d4108e3230201392bd8a9dce1ff4a62" Oct 30 17:52:47.082213 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:52:47.082184 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12868ab2ecabe7977f26aca841bda3e9d4108e3230201392bd8a9dce1ff4a62\": container with ID starting with e12868ab2ecabe7977f26aca841bda3e9d4108e3230201392bd8a9dce1ff4a62 not found: ID does not exist" containerID="e12868ab2ecabe7977f26aca841bda3e9d4108e3230201392bd8a9dce1ff4a62" Oct 30 17:52:47.082335 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.082220 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12868ab2ecabe7977f26aca841bda3e9d4108e3230201392bd8a9dce1ff4a62"} err="failed to get container status \"e12868ab2ecabe7977f26aca841bda3e9d4108e3230201392bd8a9dce1ff4a62\": rpc error: code = NotFound desc = could not find container \"e12868ab2ecabe7977f26aca841bda3e9d4108e3230201392bd8a9dce1ff4a62\": container with ID starting with e12868ab2ecabe7977f26aca841bda3e9d4108e3230201392bd8a9dce1ff4a62 not found: ID does not exist" Oct 30 17:52:47.122455 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.122420 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tmp-dir\") pod \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " Oct 30 17:52:47.122614 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.122476 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-model-cache\") pod \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " Oct 30 17:52:47.122614 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.122542 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tls-certs\") pod \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " Oct 30 17:52:47.122614 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.122577 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8940fb3-07af-4f6b-90a4-afdd309b6b69-kube-api-access-x62fr\" (UniqueName: \"kubernetes.io/projected/a8940fb3-07af-4f6b-90a4-afdd309b6b69-kube-api-access-x62fr\") pod \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " Oct 30 17:52:47.122756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.122616 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-home\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-home\") pod \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " Oct 30 17:52:47.122756 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.122636 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-dshm\") pod \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\" (UID: \"a8940fb3-07af-4f6b-90a4-afdd309b6b69\") " Oct 30 17:52:47.122881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.122807 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tmp-dir" pod "a8940fb3-07af-4f6b-90a4-afdd309b6b69" (UID: "a8940fb3-07af-4f6b-90a4-afdd309b6b69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:47.122881 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.122819 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-model-cache" pod "a8940fb3-07af-4f6b-90a4-afdd309b6b69" (UID: "a8940fb3-07af-4f6b-90a4-afdd309b6b69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:47.123042 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.122955 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-home" pod "a8940fb3-07af-4f6b-90a4-afdd309b6b69" (UID: "a8940fb3-07af-4f6b-90a4-afdd309b6b69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:47.123042 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.123030 4828 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tmp-dir\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:52:47.123150 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.123047 4828 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-model-cache\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:52:47.124815 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.124791 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-dshm" pod "a8940fb3-07af-4f6b-90a4-afdd309b6b69" (UID: "a8940fb3-07af-4f6b-90a4-afdd309b6b69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:52:47.124916 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.124823 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tls-certs" pod "a8940fb3-07af-4f6b-90a4-afdd309b6b69" (UID: "a8940fb3-07af-4f6b-90a4-afdd309b6b69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:52:47.124979 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.124959 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8940fb3-07af-4f6b-90a4-afdd309b6b69-kube-api-access-x62fr" pod "a8940fb3-07af-4f6b-90a4-afdd309b6b69" (UID: "a8940fb3-07af-4f6b-90a4-afdd309b6b69"). InnerVolumeSpecName "kube-api-access-x62fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:52:47.224217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.224177 4828 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8940fb3-07af-4f6b-90a4-afdd309b6b69-tls-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:52:47.224217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.224211 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x62fr\" (UniqueName: \"kubernetes.io/projected/a8940fb3-07af-4f6b-90a4-afdd309b6b69-kube-api-access-x62fr\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:52:47.224217 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.224222 4828 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-home\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:52:47.224473 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.224232 4828 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8940fb3-07af-4f6b-90a4-afdd309b6b69-dshm\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:52:47.421048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.421004 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l"] Oct 30 17:52:47.431614 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:47.431579 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-b0591e0c-kserve-56478bb5d9-jqh8l"] Oct 30 17:52:48.490017 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:52:48.489983 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8940fb3-07af-4f6b-90a4-afdd309b6b69" path="/var/lib/kubelet/pods/a8940fb3-07af-4f6b-90a4-afdd309b6b69/volumes" Oct 30 17:53:02.488827 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:02.488785 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:53:07.909839 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:53:07.909797 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:53:10.576076 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.576032 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-7qmhf"] Oct 30 17:53:10.576619 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.576597 4828 cpu_manager.go:414] "containerMap: removing container" podUID="e2a822dd-07c1-4ad7-8b34-f1f573199e53" containerName="main" Oct 30 17:53:10.576679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.576622 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a822dd-07c1-4ad7-8b34-f1f573199e53" containerName="main" Oct 30 17:53:10.576679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.576644 4828 cpu_manager.go:414] "containerMap: removing container" podUID="a8940fb3-07af-4f6b-90a4-afdd309b6b69" containerName="main" Oct 30 17:53:10.576679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.576651 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8940fb3-07af-4f6b-90a4-afdd309b6b69" containerName="main" Oct 30 17:53:10.576794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.576759 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a822dd-07c1-4ad7-8b34-f1f573199e53" containerName="main" Oct 30 17:53:10.576794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.576770 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8940fb3-07af-4f6b-90a4-afdd309b6b69" containerName="main" Oct 30 17:53:10.579968 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.579943 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:10.609019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.608982 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-7qmhf"] Oct 30 17:53:10.679383 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.679344 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8502ce65-e63a-4a9f-93e7-8d6c2c717589-maas-api-tls\") pod \"maas-api-595cc648c-7qmhf\" (UID: \"8502ce65-e63a-4a9f-93e7-8d6c2c717589\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:10.679564 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.679462 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8502ce65-e63a-4a9f-93e7-8d6c2c717589-metrics-certs\") pod \"maas-api-595cc648c-7qmhf\" (UID: \"8502ce65-e63a-4a9f-93e7-8d6c2c717589\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:10.679564 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.679489 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhzh\" (UniqueName: \"kubernetes.io/projected/8502ce65-e63a-4a9f-93e7-8d6c2c717589-kube-api-access-wmhzh\") pod \"maas-api-595cc648c-7qmhf\" (UID: \"8502ce65-e63a-4a9f-93e7-8d6c2c717589\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:10.679696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.679647 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8502ce65-e63a-4a9f-93e7-8d6c2c717589-openshift-service-ca\") pod \"maas-api-595cc648c-7qmhf\" (UID: \"8502ce65-e63a-4a9f-93e7-8d6c2c717589\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:10.780725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.780686 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8502ce65-e63a-4a9f-93e7-8d6c2c717589-metrics-certs\") pod \"maas-api-595cc648c-7qmhf\" (UID: \"8502ce65-e63a-4a9f-93e7-8d6c2c717589\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:10.780725 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.780732 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhzh\" (UniqueName: \"kubernetes.io/projected/8502ce65-e63a-4a9f-93e7-8d6c2c717589-kube-api-access-wmhzh\") pod \"maas-api-595cc648c-7qmhf\" (UID: \"8502ce65-e63a-4a9f-93e7-8d6c2c717589\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:10.781020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.780778 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8502ce65-e63a-4a9f-93e7-8d6c2c717589-openshift-service-ca\") pod \"maas-api-595cc648c-7qmhf\" (UID: \"8502ce65-e63a-4a9f-93e7-8d6c2c717589\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:10.781020 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.780832 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8502ce65-e63a-4a9f-93e7-8d6c2c717589-maas-api-tls\") pod \"maas-api-595cc648c-7qmhf\" (UID: \"8502ce65-e63a-4a9f-93e7-8d6c2c717589\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:10.781657 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.781632 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8502ce65-e63a-4a9f-93e7-8d6c2c717589-openshift-service-ca\") pod \"maas-api-595cc648c-7qmhf\" (UID: \"8502ce65-e63a-4a9f-93e7-8d6c2c717589\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:10.783323 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.783299 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8502ce65-e63a-4a9f-93e7-8d6c2c717589-metrics-certs\") pod \"maas-api-595cc648c-7qmhf\" (UID: \"8502ce65-e63a-4a9f-93e7-8d6c2c717589\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:10.783444 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.783376 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8502ce65-e63a-4a9f-93e7-8d6c2c717589-maas-api-tls\") pod \"maas-api-595cc648c-7qmhf\" (UID: \"8502ce65-e63a-4a9f-93e7-8d6c2c717589\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:10.804202 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.804169 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhzh\" (UniqueName: \"kubernetes.io/projected/8502ce65-e63a-4a9f-93e7-8d6c2c717589-kube-api-access-wmhzh\") pod \"maas-api-595cc648c-7qmhf\" (UID: \"8502ce65-e63a-4a9f-93e7-8d6c2c717589\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:10.894614 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:10.894509 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:11.201916 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:11.201872 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-7qmhf"] Oct 30 17:53:11.203303 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:53:11.203272 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8502ce65_e63a_4a9f_93e7_8d6c2c717589.slice/crio-0463cdca4dbca25c26f17751cad8773ac1613bed91ce2e32a8e4b4ae52b88cb6 WatchSource:0}: Error finding container 0463cdca4dbca25c26f17751cad8773ac1613bed91ce2e32a8e4b4ae52b88cb6: Status 404 returned error can't find the container with id 0463cdca4dbca25c26f17751cad8773ac1613bed91ce2e32a8e4b4ae52b88cb6 Oct 30 17:53:11.204805 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:11.204785 4828 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:53:11.228060 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:11.228019 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" event={"ID":"8502ce65-e63a-4a9f-93e7-8d6c2c717589","Type":"ContainerStarted","Data":"0463cdca4dbca25c26f17751cad8773ac1613bed91ce2e32a8e4b4ae52b88cb6"} Oct 30 17:53:12.236525 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:12.236481 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" event={"ID":"8502ce65-e63a-4a9f-93e7-8d6c2c717589","Type":"ContainerStarted","Data":"0a3dc90d7206da81847bb2191b0d3cc95ffb84d8728d6e9effaa3cf87c042879"} Oct 30 17:53:12.236970 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:12.236600 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:12.279205 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:12.279146 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" podStartSLOduration=1.921317277 podStartE2EDuration="2.279131294s" totalImagesPullingTime="357.814017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:53:12.278028155 +0000 UTC m=+3969.923538338" watchObservedRunningTime="2026-10-30 17:53:12.279131294 +0000 UTC m=+3969.924641470" Oct 30 17:53:18.248392 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:18.248356 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-7qmhf" Oct 30 17:53:21.804448 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:21.804036 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-nmv99 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:53:21.804448 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:21.804114 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-nmv99" podUID="bfa4964f-39da-49c6-bcf5-515c41e85e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:53:22.383925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:22.383098 4828 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-r26lb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:53:22.383925 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:22.383184 4828 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-r26lb" podUID="1fc9a9e7-416b-4b87-912e-a524f41e0d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:53:25.658207 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:25.658167 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp"] Oct 30 17:53:25.659265 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:25.658364 4828 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" podUID="3563a565-f8f5-44b3-b969-a14c15a09155" containerName="maas-api" containerID="cri-o://c7bf51fe92fbe0f9088934ec0cd8ca28f9143265469b6b2c89adf4b0e2ae0833" gracePeriod=30 Oct 30 17:53:25.909107 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:25.909040 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 17:53:25.930339 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:25.930312 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3563a565-f8f5-44b3-b969-a14c15a09155-kube-api-access-hbknx\" (UniqueName: \"kubernetes.io/projected/3563a565-f8f5-44b3-b969-a14c15a09155-kube-api-access-hbknx\") pod \"3563a565-f8f5-44b3-b969-a14c15a09155\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " Oct 30 17:53:25.930518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:25.930360 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3563a565-f8f5-44b3-b969-a14c15a09155-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3563a565-f8f5-44b3-b969-a14c15a09155-openshift-service-ca\") pod \"3563a565-f8f5-44b3-b969-a14c15a09155\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " Oct 30 17:53:25.930518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:25.930387 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-maas-api-tls\") pod \"3563a565-f8f5-44b3-b969-a14c15a09155\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " Oct 30 17:53:25.930518 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:25.930430 4828 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-metrics-certs\") pod \"3563a565-f8f5-44b3-b969-a14c15a09155\" (UID: \"3563a565-f8f5-44b3-b969-a14c15a09155\") " Oct 30 17:53:25.930759 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:25.930727 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3563a565-f8f5-44b3-b969-a14c15a09155-openshift-service-ca" pod "3563a565-f8f5-44b3-b969-a14c15a09155" (UID: "3563a565-f8f5-44b3-b969-a14c15a09155"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:53:25.930889 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:25.930864 4828 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3563a565-f8f5-44b3-b969-a14c15a09155-openshift-service-ca\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:53:25.932519 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:25.932494 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3563a565-f8f5-44b3-b969-a14c15a09155-kube-api-access-hbknx" pod "3563a565-f8f5-44b3-b969-a14c15a09155" (UID: "3563a565-f8f5-44b3-b969-a14c15a09155"). InnerVolumeSpecName "kube-api-access-hbknx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:53:25.932615 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:25.932524 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-maas-api-tls" pod "3563a565-f8f5-44b3-b969-a14c15a09155" (UID: "3563a565-f8f5-44b3-b969-a14c15a09155"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:25.932615 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:25.932548 4828 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-metrics-certs" pod "3563a565-f8f5-44b3-b969-a14c15a09155" (UID: "3563a565-f8f5-44b3-b969-a14c15a09155"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:53:26.031628 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:26.031584 4828 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-metrics-certs\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:53:26.031628 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:26.031623 4828 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbknx\" (UniqueName: \"kubernetes.io/projected/3563a565-f8f5-44b3-b969-a14c15a09155-kube-api-access-hbknx\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:53:26.031628 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:26.031636 4828 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3563a565-f8f5-44b3-b969-a14c15a09155-maas-api-tls\") on node \"ip-10-0-135-153.ec2.internal\" DevicePath \"\"" Oct 30 17:53:26.340978 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:26.340944 4828 generic.go:356] "Generic (PLEG): container finished" podID="3563a565-f8f5-44b3-b969-a14c15a09155" containerID="c7bf51fe92fbe0f9088934ec0cd8ca28f9143265469b6b2c89adf4b0e2ae0833" exitCode=0 Oct 30 17:53:26.341417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:26.340998 4828 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" Oct 30 17:53:26.341417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:26.341032 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" event={"ID":"3563a565-f8f5-44b3-b969-a14c15a09155","Type":"ContainerDied","Data":"c7bf51fe92fbe0f9088934ec0cd8ca28f9143265469b6b2c89adf4b0e2ae0833"} Oct 30 17:53:26.341417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:26.341098 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp" event={"ID":"3563a565-f8f5-44b3-b969-a14c15a09155","Type":"ContainerDied","Data":"7865bfcff7681d0f6fec95d75cd5b5a949b49b5e75f5838afe6121ccf9d3ba0c"} Oct 30 17:53:26.341417 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:26.341128 4828 scope.go:122] "RemoveContainer" containerID="c7bf51fe92fbe0f9088934ec0cd8ca28f9143265469b6b2c89adf4b0e2ae0833" Oct 30 17:53:26.350985 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:26.350966 4828 scope.go:122] "RemoveContainer" containerID="c7bf51fe92fbe0f9088934ec0cd8ca28f9143265469b6b2c89adf4b0e2ae0833" Oct 30 17:53:26.351259 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:53:26.351242 4828 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7bf51fe92fbe0f9088934ec0cd8ca28f9143265469b6b2c89adf4b0e2ae0833\": container with ID starting with c7bf51fe92fbe0f9088934ec0cd8ca28f9143265469b6b2c89adf4b0e2ae0833 not found: ID does not exist" containerID="c7bf51fe92fbe0f9088934ec0cd8ca28f9143265469b6b2c89adf4b0e2ae0833" Oct 30 17:53:26.351303 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:26.351269 4828 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7bf51fe92fbe0f9088934ec0cd8ca28f9143265469b6b2c89adf4b0e2ae0833"} err="failed to get container status \"c7bf51fe92fbe0f9088934ec0cd8ca28f9143265469b6b2c89adf4b0e2ae0833\": rpc error: code = NotFound desc = could not find container \"c7bf51fe92fbe0f9088934ec0cd8ca28f9143265469b6b2c89adf4b0e2ae0833\": container with ID starting with c7bf51fe92fbe0f9088934ec0cd8ca28f9143265469b6b2c89adf4b0e2ae0833 not found: ID does not exist" Oct 30 17:53:26.391959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:26.391920 4828 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp"] Oct 30 17:53:26.403482 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:26.403444 4828 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-vcmmp"] Oct 30 17:53:26.489100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:53:26.489062 4828 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3563a565-f8f5-44b3-b969-a14c15a09155" path="/var/lib/kubelet/pods/3563a565-f8f5-44b3-b969-a14c15a09155/volumes" Oct 30 17:53:37.909169 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:53:37.909122 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:54:02.594999 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:02.594956 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:54:03.209650 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:03.209614 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:54:07.912504 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:54:07.912465 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:54:20.201727 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:20.201692 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-575996c6dd-2tngr_21c11229-9597-4027-b247-2c5ad341a24b/manager/0.log" Oct 30 17:54:20.697694 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:20.697656 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-rgk8v_1f68ded2-8cca-48d6-bbba-84e747e3b6cb/manager/0.log" Oct 30 17:54:21.216828 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:21.216793 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78487d6d64-k525h_b8f04c76-eb20-4b7a-8896-e09db947337d/manager/0.log" Oct 30 17:54:22.434348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:22.434316 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68c46f97f5-k89lj_c52cbadc-73a6-4a8c-976d-9d118208b84d/copy-manifests/0.log" Oct 30 17:54:22.478143 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:22.478109 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68c46f97f5-k89lj_c52cbadc-73a6-4a8c-976d-9d118208b84d/manager/0.log" Oct 30 17:54:22.980950 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:22.980916 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-b8bbs_177c030c-d783-44ae-92e9-3368ed00ef0d/proxy/0.log" Oct 30 17:54:23.828023 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:23.827989 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-7qmhf_8502ce65-e63a-4a9f-93e7-8d6c2c717589/maas-api/0.log" Oct 30 17:54:25.955342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:25.955297 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-c9mfs_d1396d19-0a39-4d00-bbf0-ce5cd260354c/postgres/0.log" Oct 30 17:54:27.978268 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:27.978237 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-2xqwn_215337ca-676e-4628-ab89-9da3fd69ae17/manager/0.log" Oct 30 17:54:28.413590 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:28.413561 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-mrcfr_175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3/manager/0.log" Oct 30 17:54:30.127118 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:30.127077 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-768ffddf88-mg4t4_ebe8bb77-c61f-4429-a912-26e22cea3c02/kube-auth-proxy/0.log" Oct 30 17:54:30.517684 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:30.517656 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-22qj7_54ac60a7-ad93-4811-8242-5f28289f5d1c/istio-proxy/0.log" Oct 30 17:54:31.941049 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:31.941017 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-nmv99_bfa4964f-39da-49c6-bcf5-515c41e85e72/istio-proxy/0.log" Oct 30 17:54:32.161285 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:32.161252 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r26lb_1fc9a9e7-416b-4b87-912e-a524f41e0d53/istio-proxy/1.log" Oct 30 17:54:32.384006 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:32.383972 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-xmtgb_aaebd75a-5133-4ca6-b3f9-9adfd87dd67d/istio-proxy/0.log" Oct 30 17:54:32.585818 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:32.585785 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-w9xlk_a7513be4-9787-41d1-99a1-29ffc37bc75c/payload-pre-processing/0.log" Oct 30 17:54:32.784542 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:32.784511 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mj8j7_c055e279-12c7-4707-b602-56548f5df870/payload-processing/0.log" Oct 30 17:54:32.998401 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:32.998363 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c78887cdf-9jhzv_a6361ea7-1d84-4bc8-b439-3f60ed9d9e40/router/0.log" Oct 30 17:54:33.466956 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:33.466924 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r26lb_1fc9a9e7-416b-4b87-912e-a524f41e0d53/istio-proxy/0.log" Oct 30 17:54:34.040705 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:34.040674 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r_6ab66bcb-c480-4c94-abe5-307e17932f99/main/0.log" Oct 30 17:54:34.473632 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:34.473553 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-p7tp8_86d11598-b8b5-4653-93ab-438cf26e63eb/main/0.log" Oct 30 17:54:34.916960 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:34.916930 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp_204a77c5-f77b-4842-ae59-cfe3c86cd976/main/0.log" Oct 30 17:54:37.910297 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:54:37.910261 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:54:45.592822 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:45.592788 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-575996c6dd-2tngr_21c11229-9597-4027-b247-2c5ad341a24b/manager/0.log" Oct 30 17:54:46.047955 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:46.047920 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-rgk8v_1f68ded2-8cca-48d6-bbba-84e747e3b6cb/manager/0.log" Oct 30 17:54:46.522882 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:46.522848 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78487d6d64-k525h_b8f04c76-eb20-4b7a-8896-e09db947337d/manager/0.log" Oct 30 17:54:47.675842 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:47.675813 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68c46f97f5-k89lj_c52cbadc-73a6-4a8c-976d-9d118208b84d/copy-manifests/0.log" Oct 30 17:54:47.719069 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:47.719031 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68c46f97f5-k89lj_c52cbadc-73a6-4a8c-976d-9d118208b84d/manager/0.log" Oct 30 17:54:48.188658 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:48.188621 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-b8bbs_177c030c-d783-44ae-92e9-3368ed00ef0d/proxy/0.log" Oct 30 17:54:48.742348 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:48.742317 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-7qmhf_8502ce65-e63a-4a9f-93e7-8d6c2c717589/maas-api/0.log" Oct 30 17:54:50.831993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:50.831962 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-c9mfs_d1396d19-0a39-4d00-bbf0-ce5cd260354c/postgres/0.log" Oct 30 17:54:52.486750 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:52.486720 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-2xqwn_215337ca-676e-4628-ab89-9da3fd69ae17/manager/0.log" Oct 30 17:54:52.906237 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:52.906207 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-mrcfr_175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3/manager/0.log" Oct 30 17:54:54.389405 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:54.389361 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-768ffddf88-mg4t4_ebe8bb77-c61f-4429-a912-26e22cea3c02/kube-auth-proxy/0.log" Oct 30 17:54:54.783174 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:54.783142 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-22qj7_54ac60a7-ad93-4811-8242-5f28289f5d1c/istio-proxy/0.log" Oct 30 17:54:56.168413 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:56.168374 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-nmv99_bfa4964f-39da-49c6-bcf5-515c41e85e72/istio-proxy/0.log" Oct 30 17:54:56.388019 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:56.387982 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r26lb_1fc9a9e7-416b-4b87-912e-a524f41e0d53/istio-proxy/1.log" Oct 30 17:54:56.594458 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:56.594423 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-xmtgb_aaebd75a-5133-4ca6-b3f9-9adfd87dd67d/istio-proxy/0.log" Oct 30 17:54:56.785258 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:56.785230 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-w9xlk_a7513be4-9787-41d1-99a1-29ffc37bc75c/payload-pre-processing/0.log" Oct 30 17:54:56.978315 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:56.978236 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mj8j7_c055e279-12c7-4707-b602-56548f5df870/payload-processing/0.log" Oct 30 17:54:57.185121 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:57.185089 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c78887cdf-9jhzv_a6361ea7-1d84-4bc8-b439-3f60ed9d9e40/router/0.log" Oct 30 17:54:57.736696 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:57.736663 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-t7s2r_6ab66bcb-c480-4c94-abe5-307e17932f99/main/0.log" Oct 30 17:54:58.162015 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:58.161986 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-p7tp8_86d11598-b8b5-4653-93ab-438cf26e63eb/main/0.log" Oct 30 17:54:58.563582 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:54:58.563547 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b87r7vp_204a77c5-f77b-4842-ae59-cfe3c86cd976/main/0.log" Oct 30 17:55:02.714108 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:55:02.714065 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:55:03.307775 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:55:03.307730 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:55:07.910702 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:55:07.910665 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:55:29.932096 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:55:29.932011 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78487d6d64-k525h_b8f04c76-eb20-4b7a-8896-e09db947337d/manager/0.log" Oct 30 17:55:30.309517 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:55:30.309481 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78487d6d64-k525h_b8f04c76-eb20-4b7a-8896-e09db947337d/manager/0.log" Oct 30 17:55:31.032219 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:55:31.032190 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mj8j7_c055e279-12c7-4707-b602-56548f5df870/payload-processing/0.log" Oct 30 17:55:31.231837 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:55:31.231808 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mj8j7_c055e279-12c7-4707-b602-56548f5df870/payload-processing/0.log" Oct 30 17:55:37.910781 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:55:37.910744 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:56:02.855959 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:02.855885 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:03.427330 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:03.427294 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:07.911159 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:56:07.911119 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:56:11.205259 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:11.205222 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78487d6d64-k525h_b8f04c76-eb20-4b7a-8896-e09db947337d/manager/0.log" Oct 30 17:56:11.561735 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:11.561697 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78487d6d64-k525h_b8f04c76-eb20-4b7a-8896-e09db947337d/manager/0.log" Oct 30 17:56:12.250627 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.250597 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mj8j7_c055e279-12c7-4707-b602-56548f5df870/payload-processing/0.log" Oct 30 17:56:12.437360 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.437333 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mj8j7_c055e279-12c7-4707-b602-56548f5df870/payload-processing/0.log" Oct 30 17:56:12.741358 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.741319 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2tldp/must-gather-rnhqx"] Oct 30 17:56:12.741817 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.741802 4828 cpu_manager.go:414] "containerMap: removing container" podUID="3563a565-f8f5-44b3-b969-a14c15a09155" containerName="maas-api" Oct 30 17:56:12.741865 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.741818 4828 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3563a565-f8f5-44b3-b969-a14c15a09155" containerName="maas-api" Oct 30 17:56:12.741931 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.741921 4828 memory_manager.go:365] "RemoveStaleState removing state" podUID="3563a565-f8f5-44b3-b969-a14c15a09155" containerName="maas-api" Oct 30 17:56:12.745211 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.745193 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2tldp/must-gather-rnhqx" Oct 30 17:56:12.759654 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.759632 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2tldp\"/\"kube-root-ca.crt\"" Oct 30 17:56:12.770747 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.770724 4828 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2tldp\"/\"openshift-service-ca.crt\"" Oct 30 17:56:12.770858 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.770753 4828 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2tldp\"/\"default-dockercfg-t845d\"" Oct 30 17:56:12.780883 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.780860 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2tldp/must-gather-rnhqx"] Oct 30 17:56:12.790449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.790421 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:12.895308 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.895269 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b9d197e-4507-4027-9e84-9fbe10ff9c86-must-gather-output\") pod \"must-gather-rnhqx\" (UID: \"6b9d197e-4507-4027-9e84-9fbe10ff9c86\") " pod="openshift-must-gather-2tldp/must-gather-rnhqx" Oct 30 17:56:12.895308 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.895319 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8dn6\" (UniqueName: \"kubernetes.io/projected/6b9d197e-4507-4027-9e84-9fbe10ff9c86-kube-api-access-g8dn6\") pod \"must-gather-rnhqx\" (UID: \"6b9d197e-4507-4027-9e84-9fbe10ff9c86\") " pod="openshift-must-gather-2tldp/must-gather-rnhqx" Oct 30 17:56:12.941628 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.941586 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:12.996435 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.996322 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b9d197e-4507-4027-9e84-9fbe10ff9c86-must-gather-output\") pod \"must-gather-rnhqx\" (UID: \"6b9d197e-4507-4027-9e84-9fbe10ff9c86\") " pod="openshift-must-gather-2tldp/must-gather-rnhqx" Oct 30 17:56:12.996435 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.996396 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8dn6\" (UniqueName: \"kubernetes.io/projected/6b9d197e-4507-4027-9e84-9fbe10ff9c86-kube-api-access-g8dn6\") pod \"must-gather-rnhqx\" (UID: \"6b9d197e-4507-4027-9e84-9fbe10ff9c86\") " pod="openshift-must-gather-2tldp/must-gather-rnhqx" Oct 30 17:56:12.996679 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:12.996658 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6b9d197e-4507-4027-9e84-9fbe10ff9c86-must-gather-output\") pod \"must-gather-rnhqx\" (UID: \"6b9d197e-4507-4027-9e84-9fbe10ff9c86\") " pod="openshift-must-gather-2tldp/must-gather-rnhqx" Oct 30 17:56:13.024026 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:13.023993 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8dn6\" (UniqueName: \"kubernetes.io/projected/6b9d197e-4507-4027-9e84-9fbe10ff9c86-kube-api-access-g8dn6\") pod \"must-gather-rnhqx\" (UID: \"6b9d197e-4507-4027-9e84-9fbe10ff9c86\") " pod="openshift-must-gather-2tldp/must-gather-rnhqx" Oct 30 17:56:13.059993 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:13.059955 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2tldp/must-gather-rnhqx" Oct 30 17:56:13.361048 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:13.360932 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2tldp/must-gather-rnhqx"] Oct 30 17:56:13.363746 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:56:13.363711 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b9d197e_4507_4027_9e84_9fbe10ff9c86.slice/crio-e3db08c7bb2d7fe7670c9eef027395b73e15373da4707489406975f353bdd15f WatchSource:0}: Error finding container e3db08c7bb2d7fe7670c9eef027395b73e15373da4707489406975f353bdd15f: Status 404 returned error can't find the container with id e3db08c7bb2d7fe7670c9eef027395b73e15373da4707489406975f353bdd15f Oct 30 17:56:13.391337 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:13.391305 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2tldp/must-gather-rnhqx" event={"ID":"6b9d197e-4507-4027-9e84-9fbe10ff9c86","Type":"ContainerStarted","Data":"e3db08c7bb2d7fe7670c9eef027395b73e15373da4707489406975f353bdd15f"} Oct 30 17:56:14.399503 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:14.399468 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2tldp/must-gather-rnhqx" event={"ID":"6b9d197e-4507-4027-9e84-9fbe10ff9c86","Type":"ContainerStarted","Data":"7fe26086b0489f4ceb63d74c345af4a13aaba8a1c0a9b42a3160fda8c2b86b66"} Oct 30 17:56:15.415342 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:15.415303 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2tldp/must-gather-rnhqx" event={"ID":"6b9d197e-4507-4027-9e84-9fbe10ff9c86","Type":"ContainerStarted","Data":"8ea5ffb3d6126aaa79e856ea8dddf62cd06f72feaa0a9d72fab2d3bd9eb58af0"} Oct 30 17:56:15.456539 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:15.456481 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-2tldp/must-gather-rnhqx" podStartSLOduration=2.668688061 podStartE2EDuration="3.456464422s" totalImagesPullingTime="787.776361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:15.454593205 +0000 UTC m=+4153.100103385" watchObservedRunningTime="2026-10-30 17:56:15.456464422 +0000 UTC m=+4153.101974622" Oct 30 17:56:17.645762 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:17.645708 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wgxv6_ac8ed89d-9808-4bff-9d24-3e5fc34041ad/global-pull-secret-syncer/0.log" Oct 30 17:56:17.836345 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:17.836311 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fb8qf_3095c033-c4ff-42cd-9530-1be17a144a31/connection-checker/0.log" Oct 30 17:56:17.973500 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:17.973408 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-mhsj5_159cde9b-22a5-4930-9bf6-2535ab4af3d4/connection-checker/0.log" Oct 30 17:56:18.112920 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:18.111490 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-mmhr5_30001919-517f-4031-9ef4-bc0fbb43d122/connection-checker/0.log" Oct 30 17:56:18.345921 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:18.345565 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" Oct 30 17:56:18.993926 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:18.993470 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-153.ec2.internal_8216a0ebd9e778d471b8e472a27d4665/haproxy/0.log" Oct 30 17:56:25.327444 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:25.327409 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-2xqwn_215337ca-676e-4628-ab89-9da3fd69ae17/manager/0.log" Oct 30 17:56:25.596371 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:25.596290 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-mrcfr_175fb4ce-bdfb-4d1d-bd67-0b59fe27f8d3/manager/0.log" Oct 30 17:56:27.275145 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.275100 4828 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b"] Oct 30 17:56:27.279497 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.279470 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.296134 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.296107 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b"] Oct 30 17:56:27.366869 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.366831 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/556fe11d-01b2-400a-95a0-7a2882069286-lib-modules\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.366869 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.366878 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/556fe11d-01b2-400a-95a0-7a2882069286-sys\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.367100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.366972 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/556fe11d-01b2-400a-95a0-7a2882069286-podres\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.367100 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.367028 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gvkz\" (UniqueName: \"kubernetes.io/projected/556fe11d-01b2-400a-95a0-7a2882069286-kube-api-access-5gvkz\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.367177 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.367145 4828 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/556fe11d-01b2-400a-95a0-7a2882069286-proc\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.467658 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.467608 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/556fe11d-01b2-400a-95a0-7a2882069286-proc\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.467851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.467685 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/556fe11d-01b2-400a-95a0-7a2882069286-lib-modules\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.467851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.467719 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/556fe11d-01b2-400a-95a0-7a2882069286-sys\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.467851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.467738 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/556fe11d-01b2-400a-95a0-7a2882069286-proc\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.467851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.467752 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/556fe11d-01b2-400a-95a0-7a2882069286-podres\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.467851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.467799 4828 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gvkz\" (UniqueName: \"kubernetes.io/projected/556fe11d-01b2-400a-95a0-7a2882069286-kube-api-access-5gvkz\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.467851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.467821 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/556fe11d-01b2-400a-95a0-7a2882069286-podres\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.467851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.467825 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/556fe11d-01b2-400a-95a0-7a2882069286-lib-modules\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.467851 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.467827 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/556fe11d-01b2-400a-95a0-7a2882069286-sys\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.491267 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.491227 4828 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gvkz\" (UniqueName: \"kubernetes.io/projected/556fe11d-01b2-400a-95a0-7a2882069286-kube-api-access-5gvkz\") pod \"perf-node-gather-daemonset-2jd4b\" (UID: \"556fe11d-01b2-400a-95a0-7a2882069286\") " pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.597196 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.597095 4828 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:27.917204 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:27.917176 4828 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b"] Oct 30 17:56:27.919478 ip-10-0-135-153 kubenswrapper[4828]: W1030 17:56:27.919450 4828 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod556fe11d_01b2_400a_95a0_7a2882069286.slice/crio-417088b801ea84464f6dd6ceac5cd3daaeb20b79a0cf7dccd6e4216cd418266b WatchSource:0}: Error finding container 417088b801ea84464f6dd6ceac5cd3daaeb20b79a0cf7dccd6e4216cd418266b: Status 404 returned error can't find the container with id 417088b801ea84464f6dd6ceac5cd3daaeb20b79a0cf7dccd6e4216cd418266b Oct 30 17:56:28.140849 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:28.140813 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-c69x2_dbb5e2c6-44f4-4152-a7ac-98ae4c045190/cluster-monitoring-operator/0.log" Oct 30 17:56:28.555033 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:28.554993 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" event={"ID":"556fe11d-01b2-400a-95a0-7a2882069286","Type":"ContainerStarted","Data":"8ed08175b5561421bf568a7152cf3cfd87196c6a32773adc5507edda8f12cf56"} Oct 30 17:56:28.555446 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:28.555048 4828 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" event={"ID":"556fe11d-01b2-400a-95a0-7a2882069286","Type":"ContainerStarted","Data":"417088b801ea84464f6dd6ceac5cd3daaeb20b79a0cf7dccd6e4216cd418266b"} Oct 30 17:56:28.555446 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:28.555125 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:28.597752 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:28.597683 4828 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" podStartSLOduration=1.597667822 podStartE2EDuration="1.597667822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:28.594753513 +0000 UTC m=+4166.240263761" watchObservedRunningTime="2026-10-30 17:56:28.597667822 +0000 UTC m=+4166.243178001" Oct 30 17:56:29.540406 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:29.540374 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dw2pc_3ffdad02-6e93-49ac-97d7-25f5a6ed61c4/node-exporter/0.log" Oct 30 17:56:29.665589 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:29.665558 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dw2pc_3ffdad02-6e93-49ac-97d7-25f5a6ed61c4/kube-rbac-proxy/0.log" Oct 30 17:56:29.791653 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:29.791583 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dw2pc_3ffdad02-6e93-49ac-97d7-25f5a6ed61c4/init-textfile/0.log" Oct 30 17:56:33.838378 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:33.838330 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-7zrs4_7918c4b3-b4f4-4435-9de9-77fbe4906a7f/networking-console-plugin/0.log" Oct 30 17:56:34.181729 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:34.181642 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:34.335533 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:34.335489 4828 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4hlmr_9b0a1c6a-9bae-4e63-a98b-a5ff600ae63a/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:34.575345 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:34.575311 4828 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2tldp/perf-node-gather-daemonset-2jd4b" Oct 30 17:56:34.969785 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:34.969669 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/2.log" Oct 30 17:56:34.974605 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:34.974578 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-n758f_dacad254-b0c1-48fd-8377-1067d65ea230/console-operator/3.log" Oct 30 17:56:37.262315 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:37.262273 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-ggh56_d7a04182-8031-4fa8-8bed-4f8b60b6cc97/volume-data-source-validator/0.log" Oct 30 17:56:37.932846 ip-10-0-135-153 kubenswrapper[4828]: E1030 17:56:37.932795 4828 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7sxm" containerName="extractor" Oct 30 17:56:39.753581 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:39.753544 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6rrfj_f13fd623-29b9-4d8b-b76d-668ca123010f/dns/0.log" Oct 30 17:56:39.879848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:39.879811 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6rrfj_f13fd623-29b9-4d8b-b76d-668ca123010f/kube-rbac-proxy/0.log" Oct 30 17:56:40.379173 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:40.379144 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nbbqf_f8aeb454-5336-4fc1-8194-54d54a6cdb6c/dns-node-resolver/0.log" Oct 30 17:56:41.785872 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:41.785844 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hfsmw_e7b97c2d-82a0-47b2-aa06-c77c9c9ca67f/node-ca/0.log" Oct 30 17:56:42.565895 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:42.565865 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-768ffddf88-mg4t4_ebe8bb77-c61f-4429-a912-26e22cea3c02/kube-auth-proxy/0.log" Oct 30 17:56:42.709146 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:42.709112 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-22qj7_54ac60a7-ad93-4811-8242-5f28289f5d1c/istio-proxy/0.log" Oct 30 17:56:43.177813 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:43.177780 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-nmv99_bfa4964f-39da-49c6-bcf5-515c41e85e72/istio-proxy/0.log" Oct 30 17:56:43.246682 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:43.246651 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r26lb_1fc9a9e7-416b-4b87-912e-a524f41e0d53/istio-proxy/0.log" Oct 30 17:56:43.248194 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:43.248170 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-r26lb_1fc9a9e7-416b-4b87-912e-a524f41e0d53/istio-proxy/1.log" Oct 30 17:56:43.291615 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:43.291580 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-xmtgb_aaebd75a-5133-4ca6-b3f9-9adfd87dd67d/istio-proxy/0.log" Oct 30 17:56:43.358848 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:43.358816 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-w9xlk_a7513be4-9787-41d1-99a1-29ffc37bc75c/payload-pre-processing/0.log" Oct 30 17:56:43.426794 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:43.426763 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-mj8j7_c055e279-12c7-4707-b602-56548f5df870/payload-processing/0.log" Oct 30 17:56:43.506984 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:43.506883 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c78887cdf-9jhzv_a6361ea7-1d84-4bc8-b439-3f60ed9d9e40/router/0.log" Oct 30 17:56:45.732449 ip-10-0-135-153 kubenswrapper[4828]: I1030 17:56:45.732411 4828 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-768ffddf88-mg4t4_ebe8bb77-c61f-4429-a912-26e22cea3c02/kube-auth-proxy/0.log"